Why & How Should I Review Papers?
A Dialogue In Three Parts
For more advice on reviewing in HCI/Visualization that is a) by someone more senior than I, b) has a lot more practical links to follow, and c) is not a rambling multi-thousand word dialogue between imagined participants heavily cribbed from a 16th century book by Galileo, please read Niklas Elmqvist’s blog post instead. Otherwise, carry on.
Dramatis Personae
👩🔬Salviatus, a tenured professor, idealist
👩🏫Sagredus, an early career professor, cynic
👩💻Simplicius, a grad student, naif
Part I
👩💻Simplicius: I was just assigned this review, but I don’t know if I should do it. I don’t feel like I’m an expert in the field; I’m just a grad student. Plus, what if I’m too mean? I’d feel like a jerk! Or too nice? Then I’d feel like a sucker!
👩🔬Salviatus: You were assigned the review because somebody wanted to hear what you had to say. Most people don’t wake up feeling like experts; if you wait to do reviews until you feel entirely comfortable in your area, you’ll never end up doing them. Unless you feel totally unqualified (in which case by all means refuse the review), you’re sure to have interesting and important comments. Just be upfront about areas of the paper that you didn’t understand or couldn’t verify. Often these are areas of the paper where the authors could do a better job communicating their approach anyway!
👩🏫Sagredus: Sure, all of that stuff. Also, you submitted a paper to this conference, right?
👩💻Simplicius: Yes.
👩🏫Sagredus: When you submit a paper, you’re probably creating work for at least two or three reviewers, not to mention the various chairs and committee members. That work has to come from somewhere. If you submit to a conference but don’t review for that conference (and, ideally, review about two or three times more papers than you submit) then there’s an imbalance that is going to have to come due for somebody. Either a senior person is going to take on more papers than they wanted to (which means they might be grumpier or more slapdash when it comes time to review your paper), or the chairs are going to have to look farther and farther afield for reviewers (which means somebody with very little connection to your work might end up having to review your paper). It’s both a personal and communal benefit to help out if you can.
👩🔬Salviatus: There are other benefits as well. The first is that it affords you a peek into the cutting edge of research; you can see where the field is going, and what people are interested in. It’s like a crystal ball that lets you see into the field’s future. It also improves your own writing and research: by seeing work in progress or even work that will be rejected, you can see not just what works, but what didn’t work, and so look at how to improve on these aspects in your own papers. Lastly, of course, it is a way of being more involved in your community. To be a leader in a conference one ought to have served on the committee. To serve on the committee one ought to have reviewed.
👩💻Simplicius: Some of these benefits are very long term, or somewhat difficult to quantify. What are the direct benefits to my career for accepting this review, as opposed to just spending the extra time on my own work? Or, rather, what are the opportunity costs for saying “no?”
👩🔬Salviatus: Being a good citizen of the research community is its own reward. I suppose, practically, that you can add “reviewed for this conference” to your CV as well. And having more data points about what kinds of papers get accepted or rejected, and for what reasons, is directly helpful when it comes time to submit.
👩🏫Sagredus: The main lever for getting people to accept reviews is guilt. We both used this tactic to try to convince you. Salviatus asked you to be a “good citizen” just now, and I earlier asked you to consider the work that your submissions create for others. I think a lot of academia is built on using guilt to extract labor, but that’s a discussion for another day. Still, I think that reviewing is pretty much always altruistic at heart. It is hard to quantify its direct benefit to you personally. If you end up in a faculty job, some places have internal tenure-relevant metrics to force you keep up a certain level of reviewing activity, and it would certainly look very odd to an interviewer if you didn’t have any reviewing expertise in your CV, but I think it’s mostly just about altruism and the norms of whatever community you’re in.
👩💻Simplicius: Thanks to you both for your advice. I will accept this review, and submit it as soon as possible. I still have further questions, however. Why was I chosen for this review, and not another person?
👩🔬Salviatus: It is as I said. The paper chair thought deeply about the contents of the paper, and chose you, out of all of your peers, to lend your expertise.
👩🏫Sagredus: As a grad student? It could be any number of reasons: i) your advisor was assigned too many reviews, and pawned this one off on you (either directly, or indirectly by forcing the committee to look for a related reviewer who is not overburdened already). ii) the paper you’re reviewing cited one of your papers and so the committee member picked you as a person who they at least know works in a related area. iii) the committee member remembers you from a talk you gave, or a conversation with you or your advisor, and knows that you’re working in this area iv) you reviewed last year and didn’t screw up so you got put on the list of people to whom one can semi-reliably assign reviews for the foreseeable future.
👩💻Simplicius: How were you able to speak the little “i)” numbers out loud? That’s very impressive. However, I am still confused. My labmates, some of whom are more senior than I am, didn’t receive any requests. What is different about them?
👩🏫Sagredus: A lot of it is luck. If you haven’t reviewed before, then you won’t show up in most of the automated reviewer systems. If that’s the case, then unless your advisor is particularly influential in the conference, or your work gets cited or referenced enough times by the people submitting papers, you’ll be unlikely to get assigned any reviews. However, once you’re in the system, then you’ll probably get reviews indefinitely. Doing more work and writing more papers and showing up to more conferences can make those happenstances more likely, but there’s no hard and fast rule for when you get assigned your first review. There’s also a bit of “when it rains, it pours” to this process: getting assigned your first review is much harder than getting assigned your hundredth-and-first review.
👩💻Simplicius: Very well. I will not worry about it, but instead work to complete my review in a diligent manner.
Part II
👩💻Simplicius: Now that I have accepted, I must ask — how is a review completed in practice? What are the components of a review, and how are they to be assembled?
👩🔬Salviatus: My process is as follows. First, I try to make sure I am in a positive mood, or at least not overly grouchy. We are often very negative in our field, so I try to come to the paper with the expectation that I am going to accept it. A paper is innocent until proven guilty. I then begin reading the paper. I might make a few notes, but this is an initial pass to see what the general argument of the paper is, and how it is constructed. It also gives me things to focus on in depth for the next review. After this first pass, I print out the paper and read more closely. When I find issues or have questions, I write on the paper margins with a red pen. Once I have finished this pass, I type up my notes in a text document and extract major themes. I also find it helpful to restate a summary of what I feel is the main argument of the paper. This helps me structure my review, and lets the authors (and the other reviewers) know if there was some misunderstanding or miscommunication that can be clarified.
Different reviewing systems and conferences have different questions and templates. However, I find that there are commonalities in what I pick out, so if the system just has a text box labeled “the review,” my final review might look something like this:
- A summary of the paper’s argument, in my own words. Part of a paper’s argument is likely to include a claimed contribution; if so, I make sure to restate what I believe the paper’s contribution to be as well. I make sure to also say what I found valuable or well-done in the paper, regardless of my score or recommendation. No submission is entirely valueless (or, if it was, it ought to have been desk-rejected rather than sent out for review). It’s very disheartening to receive a review that is relentlessly negative, and very rude to write one that has nothing nice to say.
- My recommendation (e.g., acceptance, weak acceptance, rejection) and a brief explanation of why. This explanation ideally has two or three major themes that I enumerate and then promise to discuss in depth.
- The discussion of major themes or issues, with specifics and quotes where possible, and specific suggestions for improvement. “I found the paper’s argumentation about X to be unconvincing” and other such vagueries are not useful feedback. Ideally you’d cleanly identify issue X, and suggest that the authors perform action Y or Z to address it.
- A section for minor issues. These include things like spelling or usage errors, or fixes to figure legibility or typesetting, or other papers to consult. Things that would help the authors improve their submission, but aren’t necessarily part of your rationale for accepting or rejecting it.
👩🏫Sagredus: I mostly agree with this strategy. If you write a paper review at the same time you’re reading it for the first time, you can get hung up on very minor issues and sink the paper because you don’t have a holistic idea of what it’s doing or why. Read once, sit on it a little, then read again. But again, this is all rather idiosyncratic. Find a structure that works for you rather than just blindly copy someone else’s format. The main bits you need to get across are your rating, your argumentation for that rating, and your suggestions for what the authors should do next. I will say, though, that if you have no structure at all, your reviews can be sort of rambly. And if there’s too much structure, the reviews seem more like you ticking off check boxes than seriously engaging with the paper’s argumentation and structure, and, frankly, encourages the sort of cookie cutter papers that can stifle more creative work (although one person’s “cookie cutter template” is another person’s “pre-flight checklist to avoid disaster”).
👩💻Simplicius: Salviatus mentioned a red pen. Am I supposed to find and list out all of the grammar or usage errors in a paper?
👩🔬Salviatus: Probably not. If the paper is very poorly written (especially if it’s to the extent that it damages your ability to interpret it), then you should mention that and cite some concrete examples of course, but otherwise it’s not your job to grade the paper like it’s a student essay. You’re making an argument for the paper’s acceptance. Those minor polishing details will get fixed in time for the camera-ready.
👩🏫Sagredus: …or, at least, that’s what would happen if publishing companies used some of their money for editors and typesetters. They don’t do that, generally. Right now we pretty much have to rely on the authors and the reviewers to catch typos. So it’s more of a cost-benefit decision for you. If you’re pretty sure the paper isn’t going to get in, or is going to get in only if it’s majorly rewritten, then wasting a bunch of time finding mistakes that are likely going to be erased when the paper is resubmitted or revised is not useful for anybody. But, on the other hand, if the paper seems like a shoe-in, then finding typos that the authors might have missed can be helpful. As an author, it’s very easy to glaze over things and not have the ability to pick out errors that would be obvious if you were coming to the work fresh. That being said, detailed copy editing is almost never part of reviews in my field, and the norm is to mostly avoid it. So it’s probably not worth your time.
👩💻Simplicius: Wouldn’t it make sense to have an initial set of reviewers who determine the fit and technical merit of a paper for a particular venue, and then have separate editors who help with the formatting and copy-editing? Is it really fair that reviewers and authors have to act in both roles?
👩🏫Sagredus: Yes, that would make a lot of sense. That would, however, require that the major academic publishers actually reinvest the money they earn from unpaid expert labor in order to hire more personnel. We, as an academic community, have enough vested interest in the status quo to not be as mad about this refusal to do so as we should be.
👩💻Simplicius: Fair enough. I will do my best to dismantle the for-profit journal system. However, while I value this general advice, I am afraid there are practical components of my question that are unanswered. For instance, how long should my review be and how long should I take to write it?
👩🔬Salviatus: As long as it needs to be, of course. I tend to take 2–3 hours to write a review of a short, 8–10 page conference paper. Longer of course if it’s a journal paper, and longer still if I have a lot of trouble understanding the paper (due to either its complexity or its poor quality). In terms of length, for the structure I have listed above, I think much less than 500 words is too short. On the other hand, people don’t want to read a novel, and you have to manage your time, so more than 1500 words is too long. Imagine reading 1500 words about paper reviewing, for instance: you’d have to be quite bored to do so! Those are just rough bounds, though, based on writing in complete sentences and in my own rather verbose style.
👩🏫Sagredus: Look, this is going to be one of those things where you practice to get it right. You get used to reading papers the longer you write reviews, and you get better at picking out patterns as well. I would set aside a lot of time for your first review, much more than it would take you to do the equivalent amount of reading for, say, a class. Write a draft, and if there’s an ethical way for you to share that draft and get feedback, do so. For instance, some conferences or workshops let you “farm out” your assigned reviews (so long as you mention who actually assisted or prepared the review to the conference). When this is an option, you can ask your advisor to subcontract out a review to you, and then work together or get feedback on your reviewing style. As an aside, I think this is one of the only reasons why this review subcontracting should be done: as a learning opportunity for a newer student. Otherwise I think it’s sort of, I don’t know, icky, to accept a review and then immediately give it away.
👩💻Simplicius: If it is as you say, I will do my best.
Part III
👩💻Simplicius: I have completed my review and am about to submit it. But before I do so, what are some pitfalls or things to avoid when writing reviews? What do you dislike in reviews, either when you receive them or when you have to coordinate with them as a member of the committee?
👩🔬Salviatus: One thing I would like to see more of is maximum grades. Often conferences use a 1–5 rating system for recommendations, where a “5” is a “I would strongly argue for this paper’s acceptance.” We mentioned before that writing a review is not like grading an essay. What this means is that it’s very tempting to view that 5 (or the equivalent) as a “perfect score.” Since no paper is perfect, students will often avoid giving out 5s. I think this is a mistake. If you like the paper and want to see it in the conference, give it a 5! On the other end, I think that we are much too harsh with our negative reviews. As we mentioned, no paper is perfect, and it’s very easy to write long reviews about why you think a particular paper is bad. It’s a way of making yourself feel smart and incisive and so on, but it’s not particularly useful. If the paper has big glaring issues, it’s not going to get in, so just mention what those issues are and suggest how to fix them for the next go-around. There’s no reason to get nasty or personal or write extremely long polemics about why the submission is a sign of everything wrong with a community. Be supportive, even for work that you think doesn’t make the grade.
👩💻Simplicius: But what if I genuinely have nothing nice to say?
👩🔬Salviatus: You should always provide constructive criticism, even if you think the paper has no chance of making it into the conference. Perhaps you enjoyed the writing style? Then mention the clarity of the prose! Perhaps it’s an interesting topic? Mention that you’d like to see more work in this area!
👩🏫Sagredus: From experience, I can usually tell if the reviewer was really searching for something nice to say. “An interesting topic” or “clearly written” are the kings of that kind of language. So don’t be disingenuous. Again, if there’s really nothing worthwhile in the paper, then it really should have been desk rejected. If that’s genuinely how you feel, then make sure to tell the committee if you have the option of including confidential notes. But that’s almost never the case. There’s usually something redeemable. You should provide constructive criticism. I don’t mean “constructive criticism” as a weasel-word for nice, I mean that the authors should know what to do next, once they read your review. Is the experimental design bad? Then they should know that they will need to run new experiments. Is the technique not compelling? Then they know they need to go fishing for new examples. A negative review should give me a clear idea of what would need to happen for this work to receive a positive review from the same reviewer.
👩💻Simplicius: So what numerical score should I give bad papers?
👩🔬Salviatus: For fields with numerical scores, there’s often a straightforward translation to a recommendation. For instance, a 5 means “I would argue strongly for accepting this paper,” and a 1 means “I would argue strongly against accepting this paper. A 2 would then mean “this paper needs significant revisions before I would accept it,” and so on and so forth. Again, you are not grading an essay, you are making a recommendation. Don’t “take off a point” because you found an error. Vote, don’t grade. And make sure your review reflects your vote! If your review indicates that you would strongly argue against a paper getting in, I should see that argument in the body of the review!
👩🏫Sagredus: Since you’ve brought up ratings… look, I get it. When you read a paper, there are often good parts and bad parts. Giving out a high or low grade can feel like sticking your neck out, especially if you’re new to the field. So it’s very very tempting to give a rating of “3” (or whatever the middle score is) and trust that the other reviewers will be more senior and have more consistent positive or negative feelings. But resist, insofar as you can, this instinct. When it comes time for a committee member to build a consensus and write a summary review, that 3 gives us very little information. And since we have very few reviews to go off of, if nobody else is very opinionated, or if there’s an even split down the middle, the committee member will end up having a great deal of power over the paper’s fate. Go down to a 2 or up to a 4 if you feel like you can. If the conference has half-step ratings, even better; a 2.5 or a 3.5 are both more informative ratings than a 3. If you really can’t make up your mind and just have no idea if a paper should get in, by all means give a 3. But treat it sort of like an option of last resort.
👩💻Simplicius: But what if I’m wrong? What if I give the paper a 2 based on some issue I thought I found, but the other reviewers point out that this concern was ill-founded? If I had given the paper a 3 it might make it in, but with a 2 it’s far less likely to, even though it’s for misguided reasons.
👩🏫Sagredus: Peer reviewers are wrong all the time. It’s a very noisy process. Wrongness is only loosely correlated with, and by no means determined by, expertise. There are several places where this background wrongness can be corrected. The first is the discussion period. In many conferences, there’s a period where you can see the other reviews and comment privately to your other reviewers. This is a good time to ask questions and make sure that everybody is on the same page. The second place is in the meta or summary review. There, the committee member who is in charge of the paper can pick out the most compelling arguments from the reviews, and down-weight the things that seem ridiculous or factually incorrect. The last, of course, is the rebuttal (if the conference has one), where the authors can respond to concerns directly. But in general, an opinionated review is almost always more useful than a lukewarm review (provided you’re not just being a jerk).
👩💻Simplicius: Ah, so you’re saying there is often the ability to calibrate reviews with the other reviewers. I can always start with a high or low score based on my initial impressions and then see what the others think and adjust my score from there.
👩🔬Salviatus: You could, certainly. But there’s a problem with that tactic. And that is that many reviewers think the same way. So what happens is that you will see all of the reviewers adjust their scores towards the mean (usually a 3) in the discussion phase. The negative people will think they are being too harsh and so move up, and the positive people will have discovered new concerns and so move down. This means we sort of end up where we started, without a firm recommendation on what to do with the paper. Unless you think you got something totally wrong in a way that totally flips your opinion of the paper, it’s probably more useful to mention that you discussed things but otherwise stand your ground.
👩🏫Sagredus: Hmm, I don’t know about that. There’s already a lot of noise in reviewing. An experiment at the NeuroIPs conference found that two committees looking at identical papers disagreed with each other more than half the time. While this could be modeled by the sort of “messy middle” phenomena that Salviatus describes — good papers are good and almost always accepted, bad papers are bad and almost always rejected, and everything in the middle is a crapshoot — it could also mean that the process itself is random from the top to bottom. Almost everybody has a story about a paper that got rejected the first time it was submitted, only to be accepted or even nominated for a best paper the second time around. If the other reviewers makes you reconsider your rating, then move your rating (and indicate why in the review). Don’t worry about how hard or easy it’s going to make the committee’s job.
👩💻Simplicius: I will keep a steady heart and an open mind when reviewing. Is there anything else to keep in mind?
👩🏫Sagredus: Yep. We haven’t even gotten into some more of my pet peeves. For instance, the insistence on certain people giving feedback like “a native English speaker should review this document” which I think says much more about the reviewer and their preconceptions than it functions as helpful advice. And of course there’s the issue of how to fix the numerous glaring issues with the peer review system (for instance whether we should make people sign their reviews, and/or publish reviews alongside the paper, or, even, heaven forbid, pay reviewers for their labor). And there’s what to do about “hot topics” in research like machine learning where the number of submissions is absolutely dwarfing the amount of reviewer expertise and time. And…
👩🔬Salviatus: But for now you needn’t worry about those issues.
👩💻Simplicius: I will ignore these existential quandaries for now, and focus dutifully on my own progression as a scholar.