Reviewing a review*

Jun 26 2012 Published by under Uncategorized

While I'm not one of those people who get bombarded with dozens of review requests a month, I do enjoy what you might call a steady trickle. Since it's not yet gotten to the point of being overwhelming or productivity-quashing, I pretty much always say yes to anything within my general purview.

Now, reviewing basic science manuscripts is relatively straightforward, right?  Did the authors do all the right controls? Are the stats appropriate? Are the findings interesting and reasonably interpreted? Etc, etc. But when it comes to reviewing review papers, I find that beyond general organization and blatant wrongness, I'm not sure where to focus my well-honed critical eye.

Before the one currently in the works, I'd only reviewed two review papers in my limited but illustrious career as a peer reviewer. The first was such an utter decimation of the English language that I had to reject outright due to inability to evaluate jack squat, and the second was so ridiculously awesome that I accepted it with much vigor and enthusiasm. I've written three first-author reviews in the last 6 years or so, and all were accepted with no or very minor revisions.

This sort of thing almost never happens to me when I review (or write!) research manuscripts, and it makes me wonder whether the review process for reviews is more commonly an all-or-nothing situation. What's your experience been? What are you generally looking for when you review a review? Have you ever reviewed (or written) a review that ended up in multiple rounds of revisions?

*Sorry this is such a boring post. My brain has been swimming will all sorts of crazy thoughts this summer, and it's been hard to organize them into something coherent and of more than homeopathic levels of value.

10 responses so far

  • Zen Faulkes says:

    I've been told by editors that the standards for review articles are higher than regular articles, because reviews tend to be read and cited more. There's a greater responsibility there than usual.

  • One of my 2 reviews was barely tickled in review, one went through 3 rounds with 3 reviewers. For the latter (at a good IF journal), we had a variety of comments, and I found the process similar to that of research articles, albeit at the constructive end of the scale where the reviewers are probably OK with you publishing it in the end, but would like you to put up a fight first. The difference was heavy editor involvement, in a way I've never seen with a research article. I suppose that might bear out the first comment.

    Taking the baton from that experience, I reviewed a review recently and tried to make sure the ideas in it were better discussed and a few more alternatives considered, even though in general I was pretty happy with it from the start. So the process was more discussion and less policing, perhaps. I've also rejected a review outright for similar reasons to you. It started with a rogue apostrophe in the first sentence of the abstract and went downhill from there.

  • I only agree to review invited review articles for high-profile journals, so the authors have been pre-selected by the editorial staff to be both accomplished and (hopefully) decent writers. Given that, I see my role as a reviewer as three-fold: (1) to ensure that the authors have not ignored important work in the area that should be addressed; (2) to ensure that the authors have not inaccurately described or misrepresented work they are addressing; and (3) to encourage any restructuring or other fundamental alterations that would make the review more interesting to read.

  • I've been told by editors that the standards for review articles are higher than regular articles, because reviews tend to be read and cited more. There's a greater responsibility there than usual.

    And BTW, this is ridiculous. The editorial standards for review and research articles are completely orthogonal, and neither is "higher" or "lower" than the other.

  • qaz says:

    I rarely find that reviews are "all-or-none" things. Both in my reviewing of reviews (as a reviewer) and in my writing of reviews (as a reviewee), I find that reviews are almost always at least a one-return cycle process.

    It is very important that a review include all appropriate sides of the literature and that it define terms (and explain their consequences) correctly. And, of course, it is important that appropriate attribution be given for both experiments and theories.

    If anything, I'm harsher on review papers than on experimental papers because they tend to be the go-to papers in the literature. (People read the summaries in the reviews and rarely go back to the original. This means it is very important that reviews not misrepresent the data or its implications. For experimental papers, I'm more concerned that the experiments have been done correctly and the interpretations are not overstated.)

    Personally, I don't find a difference between high-IF and low-IF journals for reviews. And I don't care if the person is some high-muckety-muck-alpha-dog or some new graduate-student. In general, I don't review a paper unless I know the literature and the data. If the new graduate-student gets the stuff right, then great. If not, then reject. (I was once told by a senior person in my field that [as a new graduate student], I didn't have the "right" to write the review I had asked him to comment on. Needless-to-say, I ignored that comment, and it has been one of my most cited papers.)

    Of course, real "reviews" (X is true, Y is true, Z is true) are boring and mostly useless. Good reviews are really "theory" papers that bring together new concepts and integrate literature into a new perspective. At which point, we're back in "Are they likely to be right?" territory.

  • Pascale says:

    What the Comrade said: make sure all appropriate literature is cited; make sure that which is cited is correct; and make sure that the language has not been strangled. The latter can be more challenging than you think, even with a high profile senior author in a high IF journal since "the next review article" on a given topic often gets handed to very junior people. Why? It's very difficult to summarize the state of a field "with a fresh view" if you have been doing it over and over again. It's also a great way for the new kid in the lab to get up to speed on the area.

  • Oh, sweet Jesus. This takes me back.

    My postdoc PI and I (which means me alone) reviewed a fancy invited review for Neuron a while ago. It was an absolute clusterfuck.

    It was clear that the editors invited these guys to write a review because the topic was kind of sexy and not something you read much about in mainstream neuroscience journals. Having reviewed this godawful review, it became quite clear why the research in this field tends not to end up in quality neuro journals.

    Not only was the review poorly written, it was full of utterly preposterous logical fallacies (Philosophy 101 stuff -- did you know that correlation and causation are different things!?!? Also, I had to tell the authors that, just because all Xs are Ys, it does not follow that all Ys are Xs. It went on like this forever.)

    We went back and forth with these guys for probably a year or more, and it got to the point where the editors were prepared to reject the manuscript after three or four rounds of revisions, because the authors were really resisting the major issues we were bringing up. Finally, the editors threw down the gauntlet and said that the authors needed to take our suggestions for a rewrite or that was going to be the end of it. So, they rewrote it, and it ended up getting published in vastly improved form (but still pretty crappy science for Neuron, if you ask me).

    Anyway, hopefully the editors learned that, just because some dudes are at the top of their sexy-sounding subfield that is outside of mainstream neuroscience, you still probably shouldn't touch them with a ten foot pole.

  • Bashir says:

    I wrote a review paper in graduate school (an extension of my prelims). It was actually a good experience, other than the looooong time to review. I think it was a good theoretical contribution, but in a somewhat niche area. I learned a lot about how to integrate a diverse set of results into a larger theoretical point. A useful skill.

  • This has been enormously helpful. Thanks to all commentators.

    It seems like doing a really good review of a review requires much more extensive knowledge of the field and a very careful reading, compared to a regular paper.

    One question that came up for me that people here didn't address: there are different styles of review papers. One style that is problematic, but widely accepted, is the review of an author's own work. It seems to me that such a paper is not trying to present a balanced view of the field, but rather to showcase or highlight a particular research direction so that people do not need to read it in fragments. Do you feel that such a review paper needs to be help to the same standards of balanced citation and presentation as a more standard review?

Leave a Reply