Paper reviewing is one of those weird aspects of professional development that academics are rarely taught to do. Most of us are literally tossed into the deep end when we receive our first reviewer invitation. This may be the first time we’ve ever seen a manuscript that isn’t our own or someone in our lab’s. Because of the confidentiality around paper reviewing, we’re not supposed to share the manuscripts we’re reviewing, though many PI’s will start farming reviews out to grad students as “professional development.” While this practive is technically unethical, I was grateful that I had the opportunity to weigh in on a couple of manuscripts before I had to do this myself.
When I reviewed my first paper (as a senior PhD student), I was nervous. I’d dealt with my own reviews which varied widely (and still do) in terms of quality, length, and tone (I’ll get back to that). We often joke ruefully about “Reviewer # 3,” a fictional aggressive reviewer who is so contrary or aggressive that they sink a paper, even when the other two reviewers may be positive. When reading those reviews, it’s impossible not to try to guess who they’re from. Sometimes, the tone is so strident and aggressive that, despite myself, I picture a Silverback — a senior male, close to retirement, delighting in shredding up others’ work with scathing phrases like “leaps of logic” or “laughable” or “woefully inadequate.”
Other times, the review is so complementary and timid, the language gently suggestive rather than pedantic, the tone just so darned nice that you’re not getting anything useful out of it (or, worse, the feedback is useful but Reviewer #3’s aggression just barrels it right over). For those reviewers, I often find myself picturing a young woman, tentatively typing out what she hopes isn’t too mean of a review before signing off to sit, beleagured, in a faculty meeting where the SIlverback reviewer is already dominating the discussion.
And yeah, that’s some crappy internalized sexism on my part: picturing a timid mousey early-career woman and a strong, aggressive senior dude. I mean, there are all sorts of ways in which men and women are socialized differently that likely play out in reviewing. Certainly, the ways in which men and women academics participate in audience discussions (or don’t) can often play out along similar lines, and there are bullies everywhere in Academe (not just the locker room). So when I sit down to a review, I’ve tried to conquer sexist tropes and my own imposter syndrome and be a strong, useful reviewer. Sometimes, I worry I may over-correct, and be a little too aggressive (though this has a lot to do with tone and text). As a writer, have stood up to Reviewer # 3 when I feel like he’s wrong (again — why is Reviewer #3 always a he?!), as I often do, and editors have had my back, fortunately.
I’ve now reviewed dozens of manuscripts, and it’s still a process. I still doubt and tone-check myself. I always read the other reviews on a manuscript if I can, to see what I’m missing and give my own rfeview context. I try to highlight the positives while being firm about the things I feel the paper needs. When I first started, I spent way too much time on grammar, in part because I care about writing but mostly because I felt more confident about that than my opinion as a scientist. And as soon as I finally did start to feel confident, I became an editor of a couple of journals, so now I’m facing that old imposter specter all over again.
The worst, for me, is knowing when to reject. I know that rejection rates for most journals are high, but I often find it hard to bring myself to put that big red -R- on a paper. I usually opt for “Accept with Major Revisions” or “Revise and Resubmit” rather than an all-out rejection. I have no sense how often I should be rejecting papers. Is this me being nice? Mousey? Or just not-cutthroat?
Now that I have graduate students of my own, I’ve been thinking about how to train them to be reviewers, without creating either mice* or attack dogs**. If Reviewer #1 is that ineffectual, unconfident reviewer, and Reviewer #3 is too angry and aggressive, I’d like to be just right***: Reviewer #2. and We don’t usually have formal training for reviewers, in part for the reasons I mentioned about (you can’t exactly share manuscripts with your lab). I think one thing that was helpful for me was that my advisor occasionally shared his own manuscripts and reviews with us. I’m thinking of incorporating peer review exercises into my graduate classes, which was also helpful for me as a student.
Still, nothing quite prepares you for participating in the real thing — both dealing with reviews and writing them– like actually doing it. In the sciences, women are disproportionately less likely to be asked to review, which means we’re getting less practice, among other things. I’d love to see someone tackle whether my own sexist assumptions about attack dogs and mice are really reflecting something real about how we’re socialized as women to be nicer, more personable, and less aggressive (and vice-versa for men), which could play out in how we review and how those reviews are integrated into the peer review process.
How did you learn to review? Is this something you struggle(d) with? Have you figured out strategies for training your students? Are there gendered reviewing styles? I’d love to hear from you!
*Given that mice are known to eat their own babies, this may not be the best academic metaphor. Or maybe it is. Your call.
** The article uses the language “pit bull,” but see my comment below.
*** Does this make me Mama Bear?
Update: A reader rightly pointed out that the article linked here unfairly perpetuates a stereotype of pit bulls as aggressive, one which results in a lot of stigma and unfair treatment of animals. I’ve edited the wording instead to “attack dog,” to keep with the familiar metaphor (and the language of the paper I linked to) but to point towards the intentional training by humans that goes into aggressive behavior.
Tenure, She Wrote 
Having seen a lot of reviews from the other side (editor/study-section-member), where I know who the reviewer is, I was surprised to find that actually, nasty reviewer #3 tends to be young profs, postdocs, and students, terrified of “being too easy” and “missing something important”, while the “whatev” reviews are often senior faculty who don’t really care about lesser beings fouling up the literature.
Nevertheless, I agree, you want to be the good reviewer #2 – honest and fair. And sometimes, the fair thing to say is “no.”
PS. Farming reviews out to graduate students is not unethical at all. In fact, it is good practice. You MUST get permission from the editor (because otherwise you are breaking confidentiality) and you MUST retain “final responsibility” for the review (because otherwise someone else is using your name). What I do is tell the editor that I will accept the review if I can co-write it with my student XYZ. Student XYZ writes the review. I read the paper and the review and we edit the review together.
I have also heard that Reviewer #3 is very often junior people in the field (grad students, postdocs, and early career professors). I’m willing to believe it. Graduate students and junior professionals often feel like they (we?) have something to prove, and sometimes that can manifest itself in worriedly trying to ferret out all of the mistakes – lest someone judge us for missing something.
Ditto, roseofjuly. I was going to comment the same thing.
I think that many of us (including myself) are guilty of that as youngsters. In my separate comment I mentioned this same point – as a grad student I felt the need to identify every single mistake. However, I soon realized that a page-long review was ridiculous and I started focusing on only a handful of what I felt were the most important issues. Worked well for me so far and based on author responses as well as editorial actions, I feel that this has been a good strategy to follow. Nowadays, despite being pretty junior I am more often than not reviewer #2.
Yes, thanks for clarifying that. I’ve seen folks write about the review-farming from the perspective that passing of a student’s review as your own is unethical, and even if you get editor approval it still may violate confidentiality at a basic level. I’ll definitely be talking to editors about that option when the time comes.
I took a course in which 1st and 2nd year PhD students were assigned one of a few published papers to review. The topics of the published papers chosen aligned with the course content. Some guidance by the instructor, a few helpful articles that exist about writing manuscript reviews, and maybe even examples of reviews the instructor received (I can’t remember), were helpful in completing the assignment.
The instructor’s evaluation of our assignment dealt with how well we integrated course content (where applicable) and the style and substance of our review. The day we received our graded assignment back from the instructor (with comments), we met in a group with other students who reviewed the same article to discuss the things we wrote about in our individual reviews.
I found this to be a very useful exercise and I referred back to those class notes when I began writing actual reviews for journals.
Thanks for writing this, it’s a really important issue! I was a little surprised by this sentence: “I always read the other reviews on a manuscript if I can, to see what I’m missing and give my own review context.” Do you read before or after you submit your review? I consciously *don’t* look at other reviews until after mine is submitted. Partly this is because I think it better maintains a somewhat more objective form of peer review (i.e. not letting myself be too swayed by what other people think), and partly because it helps me feel more confident to write a full review on my own. I do often read the other reviews afterwards, to see how much my comments aligned and whether there were other important points I missed that I should think about next time.
In terms of training, some suggestions. A couple of years ago, EOS published a great piece on writing a solid peer review, available free here: http://onlinelibrary.wiley.com/doi/10.1029/2011EO280001/abstract
It’s not field specific and I highly recommend it.
Second, following on from the previous commenter, a lot of fields now have journals with open online discussions. For example, in the copernicus publications (I’m most familiar with Atmospheric Chemistry & Physics), a “Discussion” version is posted, as are the anonymous peer reviews, followed by the final article (and the discussion stays available even after the final version is posted). It could be a great PhD training course to picks some of these papers and have students review the discussion version, then read the actual posted peer reviews…
I always read after I submit, not before. I’m also afraid, like you, of biasing myself based on the comments of other authors.
I used to joke that if there were 3 reviewers, one would be complimentary, one would be critical (the extent of the criticism of course ‘depended’) and one would helpfully correct all your grammar. Not sure how they were numbered. Seems it goes beyond a joke????? Funny thing though, if there were only 2 then the detailed grammar was left uncorrected – yet the reviewers do not know whom else is reviewing …. hmmm… Law of Nature???
Agree with qaz that younger people, in general, tend to be more harsh – whether they are reviewers or TAs in your lab classes. People mellow as they get older and/or get more experience and tolerance of others’ inadequacies, i think. (That includes me, I have noticed over the years). But this also depends on whether the reviewer knows who wrote the paper – this can sometimes influence the more alpha-male types, perhaps. I have long thought that the authors should be anonymous and the reviewers revealed – the opposite to what usually happens.
Agree with Acclimatrix that it is very difficult to know when to ‘reject’ … if possible I always like to give people a chance to resubmit with revisions – then the choice is up to them. This is harder when you are accepting abstracts for a conference and you have a limit and just have to reject some.
I learned how to review by getting asked to review a paper for a journal. In other words, I was thrown into the deep end, lol. It was also a very prestigious multidisciplinary journal, so I was kind of in disbelief that I was asked in the first place.
One of my advisors did occasionally ask for my help reviewing grants for his NSF study section. That was very enjoyable, which was funny to me because I don’t like to write grants.
The only scientific study into this that I’m aware of (through my own limited reading, rather than anything else) suggests that female post-docs are the harshest critics (compared to UG and PG students – not enough data for more senior types though).
Male referees are more likely to accept a paper if the first author is a female (testing the same paper with male or female author names). Figs. 3 & 5 here are nice clear illustrations.
As to your second point, there are quite a few studies of gender bias in the review process, but they typically focus on the outcome for the reviewees (i.e., double-blind reviewing has been shown to increase female acceptance rates in a number of studies). There’s also that one study on CV evaluation that found that both male and female supervisors ranked competency lower when CV’s had female names rather than male (also by testing the same paper with male or female author names). So, there are some studies that contradict the finding that female authors are more likely to be accepted. But the the anecdata in the comments here also support the idea that the harshest reviewers are early career folks (though as to gender, I think we need more data).
Hi Acclimatrix – while I totally agree with you about the evidence for gender bias in evaluating CVs (it was a PNAS paper, right?), I’d genuinely love to know which studies you’re referring to for evidence of effects of double-blind reviewing on acceptance rates, as the only one(s) I know are controversial at best (e.g., Budden et al 2008a,b, TREE; but see Whittaker et al 2008, TREE), and most likely, in my opinion, have incorrect interpretations (Webb et al 2008, TREE).
First, I want to be clear that this isn’t a post about gender bias in peer review. I was referring to the studies you mentioned, and some others I don’t have with me at the moment, but I’ve also seen critiques that suggest that gender bias in peer review is no longer a major factor. Rather, I was referring in my post to the gendered ways in which we may review papers. I’ve never had a paper rejected, for example, though I have had my share of pitbull-type reviews, and I was thinking about my own reviewing, as well.
I know that there are systematic biases in terms of grants awarded (though whether it’s to do with reviewer bias or some other factor, like communication style, I don’t know), and with editors (see Nature’s recent editorial on its own biased reviewing practices), as well as a host of other areas, but that’s not what this post is about. If anything, I’d love to see more deep, systematic investigation of reviewing and gender– not from the perspective of bias against the reviewed, but in terms of training, internalized sexism like I talk about in this post, and other factors that influence how we write reviews.
Sure – sorry for dragging things off-topic.
The field of reviewing is directly fascinating to us as publishing researchers – even though the practice of analysing it remains fraught with ethical problems – asking referees to give up their limited time in properly controlled, blinded studies of their behaviour is tricky, but Sokal’s hoax is definitely another interesting and instructive example!
If anything, I’d be really interested to see if reviewing habits were heritable. Is it enough to ask those lazy/sloppy reviewers to train the next generation? What more can journals do (beyond what’s been mentioned here) to develop good reviewer habits? Do they need to be rewarded (like the Peerage of Science system where referees actually grade each others reviews)? Lots of fun questions to be asked!
No worries! I just mostly wasn’t prepared to open that as a can of worms and be able to do the conversation justice! It’s definitely a conversation worth having, and I’m happy to see that reviewer bias (of the reviewed) seems not to be the problem that it once was. It means we’re doing something right.
Great post! My PhD advisor & I did a couple of reviews together when I was a student, which was enormously helpful. He told me not to worry too much about the Introduction & Discussion sections, because that is just the framing of the paper, which can go a lot of different ways depending on the story the authors want to tell about their data. He said to really focus on the Methods & Results – if it seemed like there were major flaws there, that was what mattered. And in terms of rejecting, he told me to ask myself the question: “This might not be perfect, but does that mean it should NOT be published?” (That depends on the prestige of the journal too, obviously). In terms of tone, I also imagine that I have to sign my name to the review. That helps to keep me from getting too snarky!
My Ph.D program included discussion groups of 8-10 students + 1 faculty member in several of the major courses that all of the students had to pass through. The coursemasters deliberately chose papers that were usually good but not flawless (think PNAS direct submissions) to facilitate a good discussion, and the faculty leader helped us thoroughly review the paper and come up with a reject/revise/accept recommendation based on the big picture. I wish this were more common!
Also–for readers with trainees who need to learn how to review papers, some journals are starting to publish referee’s reviews also. EMBO Journals are a good place to start for scientists–usually pretty good papers, but often the reviewers requested some pretty substantial work for the MS to be accepted.
I am a male reviewer. I started learning, as you and probably many others in the academy did, by reviewing manuscripts that my PI sent my way when I was a grad student. Almost immediately after graduating I started getting review requests directly so I’ve reviewed plenty of manuscripts over the years. My style is to start with a 3-4 line summary of what the authors presented and mention the positives of the manuscript. I then start a second paragraph and mention only my top 2-3 negatives/improvements-needed followed by a brief 2-3 line summary of my overall thoughts. As a fresh reviewer I felt the need to jot down all the points that I could think of (I’m sure my PI took only the most relevant points) but after a few reviews I realized that its best to just focus on the most glaring issues that need to be addressed. The most satisfying experience with this strategy is when I see that the other reviewers also noted the points as me, which reinforces my feeling of having identified important points that will truly improve the manuscript.
Good news – that’s the job of the editor, not yours. if you’re not sure, lay out your concerns to the editor, so they can make an informed decision.
Right, though I am an editor now myself. 🙂
Yup–don’t worry too much about what decision to recommend, because that’s the editor’s job, and a good editor will care more about your detailed comments than your decision recommendation. If you’re not sure what decision to recommend, you can also just explain your uncertainty to the editor. Editors always appreciate it when reviewers give them context as to where they (the reviewers) are coming from.
My own advice on how to peer review a paper is here: http://dynamicecology.wordpress.com/2013/01/03/advice-how-to-review-a-manuscript-for-a-journal/. But there are increasing numbers of good sources of advice out there. I hear the British Ecological Society’s peer review manual is very good.
Thanks! I may not have made it clear in my post, but I am a subject editor now myself, so I am making rejection decisions. But even as a reviewer, most journals I review for have a recommendation button.
Thanks for the resource suggestions!
One thing that I really appreciate from the editor’s side is a no holds barred statement of what the reviewer thinks in the ‘confidential comments to the editor’ box. Not enough reviewers use that option – and it can be tricky sometimes to parse word choices that a reviewer writes in the comments to authors. As you point out, it’s easy to come off as overly harsh or overly timid without intending to.
Also +1 on all the comments about postdocs and grad students being the harshest reviewers.
Pingback: Somewhere else, part 169 | Freakonometrics
When I was in my first semester of graduate school, my PhD advisor threw me into the deep end and asked me (and a PhD candidate very close to graduation, also in the lab) to help with a review. It was way over my head at the time, but I really appreciate that he involved me so early. Plus, I was able to listen to what he and the more senior grad student had to say without saying very much myself. Another time, we were BOTH asked to review the same paper. I wasn’t telling him any details about the paper, but just told him the journal and then he said that he’d been asked to review it as well. What a great learning experience! By that point, I was far along in graduate school and we both completed our own reviews, but talked about them before we pressed submit. I am so grateful for the experiences he provided me with.
I had an advisor who worked as an editor for a medium-sized peer reviewed journal in my field. He claimed that grad students, not senior profs, gave the most scathing reviews. He thought this was due to overly high standards for what constitutes publishable work, youthful exuberance, and a degree of competitiveness that those who already had TT positions no longer have.
Pingback: Recommended reads #37 | Small Pond Science
Pingback: How Not to Be Reviewer #2 | Ashley ML Brown
Reblogged this on Grendel's (M)Others.