<ol>
<li><p>To the best of your knowledge, does the paper make a significant, well-presented, and correct contribution to the state of the art?  If the paper fails any of the three criteria, it's fair to reject it for that reason alone, regardless of the other two.</p></li>
<li><p>Here's what I think a report should contain.  Everything should be visible to the author, except <em>possibly</em> for serious accusations of misconduct.</p>

<p>a. A quick summary of the paper, to help the editor judge the quality of the results, and to help convince both the author and the editor that you actually read and understood the paper.  Place the result in its larger context.  Include a history of prior versions, even if the authors include it in the submission.  Be respectful, but brutally honest.</p>

<p>b. A discussion of the strengths and weaknesses of the paper, in terms of correctness, novelty, clarity, importance, generality, potential impact, elegance, technical depth, robustness, etc.  If you suspect unethical behavior (plagiarism, parallel submission, cooked data), describe your suspicions.  Be respectful, but brutally honest.</p>

<p>c. A recommendation to the editor for further action — accept, accept with minor revision, ask for a second round of reviewing, or reject outright.  Keep in mind that you are making a recommendation, not a decision; if you can't make up your mind, just say so.  Be respectful, but brutally honest.</p>

<p>d. More detailed feedback to the author — more detailed justification for your recommendation, requests for clarification in the final version, missing references, bugs in the proofs, simplifications, generalizations, typos, etc.  Be respectful, but brutally honest.</p></li>
<li><p>Conference reports should be shorter; program committees have hundreds of papers to consider at once.  Whether there should be a difference between conference and journal papers is up to the journal (and indirectly, up to the community).  Most theoretical computer science journals do <em>not</em> insist on a significant difference; it is quite common for the conference and journal versions of a theory paper to be essentially identical.  When in doubt, ask the editor!</p></li>
<li><p>If you still don't understand the paper after making a good-faith effort, it's the author's fault, or possibly the editor's, but <em>certainly not</em> yours.  The author's primary responsibility is to effectively communicate their result to their audience, and a good editor will send you a paper to referee only if they think you're a good representative of the paper's intended audience.  But you do have to make a good-faith effort; do not expect to immediately understand everything (anything?) immediately on your first reading.</p></li>
<li><p>If there are a <em>lot</em> of errors, don't even read the paper; just recommend rejection on the grounds that the paper is not professionally written.  Otherwise, if you really want to be thorough, include a <em>representative</em> list of grammar, spelling, and punctuation mistakes, but don't knock yourself out finding every last bug.  Be respectful, but brutally honest.</p></li>
<li><p>Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques.  Be pleasantly surprised when it doesn't actually take that long.  (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow.  Don't confuse these three possibilities.) </p></li>
<li><p>You should write at least as many referee reports as other people write for you.  If this takes more time than writing your own papers, you're not spending enough time on your own papers.</p></li>
</ol>

CC BY-SA 4.0
2018-05-29T23:58:05.240Z
My favorite part is "If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. **Don't confuse these three possibilities.**"
Caleb Stanford
CC BY-SA 2.5
2010-10-03T23:27:13.223Z
Nice punchy summary! I especially like 2a-d.
RJK
CC BY-SA 2.5
2010-10-03T21:22:07.767Z
2010-10-03T21:22:07.767Z
Jeffε
<p>A lot depends upon the conference/journal, as each community has developed its own style, so knowing what's expected of the conference/journal certainly will help a little. </p>

<blockquote>
  <p><strong>1.</strong> What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?</p>
</blockquote>

<p>Criteria: Novelty/originality, expected impact, correctness/validity, extensiveness (how much of the problem is studied? + theorems? + implementation? +experimental results?), quality of presentation. The criteria for journal acceptance are much higher than for conferences. For a conference you give a score based on the previous criteria. Verifying correctness as much as is possible is very important, especially for journal articles.</p>

<blockquote>
  <p><strong>2.</strong> What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?</p>
</blockquote>

<p>The referees report contains at least 4 main elements: A summary of the paper and its contributions; points in favour of accepting the paper; points against the paper; major comments, including points to be addressed; minor comments (typos etc). You should always give an indication of acceptance or rejection or the degree of revision required. Comments to the editor could include: a short, perhaps blunt assessment of the paper; any statement of doubt; details of possible plagiarism or parallel submission of the paper; ...</p>

<blockquote>
  <p><strong>3.</strong> How does assessment for conferences differ from those in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?</p>
</blockquote>

<p>Journal reviews tend to be more extensive and can include a lot of things for the authors
to do to bring their paper into an acceptable state, such as "implement and evaluate these ideas". Your confidence should be based on how confident you feel. Generally this will come with experience, so start out being conservative. It it's out of your field, even though you may understand the paper, being conservative is also a good idea –– it's difficult to assess originality of a paper outside of your field. Some journals say that journal submissions must contain a substantial amount of new material compared to the conference version. 30% is a figure I've heard.</p>

<blockquote>
  <p><strong>4.</strong> What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)</p>
</blockquote>

<p>This can vary. Sometimes it will be your fault, sometimes it will be theirs. Use your judgement. Maybe ask a colleague to take a look at the paper. If you totally cannot understand it and it's not due to poor formatting, then perhaps contact the PC chair and explain that this is the case. In any case, this should be reflected in your confidence score. If the paper cannot be understood due to poor writing or language or because it has been prematurely submitted, then this should be written in your report.</p>

<blockquote>
  <p><strong>5.</strong> What about typographical/grammatical mistakes? What if there are a lot of them?</p>
</blockquote>

<p>When I was younger I'd report every single typo and grammatical error. Now I just don't have the time. Reporting some is always helpful: pick the most serious ones. Also recommend that the paper be proofread (by a native speaker). For journal papers be more thorough. If it has way too many errors, then is shouldn't have been submitted to a journal, which alone is reason for rejection (IMHO).</p>

<blockquote>
  <p><strong>6.</strong> How much time should I spend on a report?</p>
</blockquote>

<p>One day maximum for a conference paper, including reading time. For journal papers, especially long ones, reading the paper carefully might take a whole week.</p>

<blockquote>
  <p><strong>7.</strong> How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?</p>
</blockquote>

<p>The expected amount could vary between 5 and 30. As a junior researcher, you'll be given a few for practice. As you advance through the grades, you'll take on more reviewing as you become a member of PCs. Then as you advance even further, you'll have an army of PhD students to do your reviewing for you. More than 30 in a year is quite onerous.</p>

<p>As I said above, I'd say it is appropriate to refuse a refereeing request when either (1) you do not feel that you have sufficient expertise to do a good review, or (2) you have too many reviewing (or other) commitments at the moment. It is also considered good form to suggest a number of possible alternative reviewers, in the case that you refuse.</p>

CC BY-SA 2.5
2010-10-03T17:27:58.097Z
2010-10-04T10:54:05.150Z
2010-10-04T10:54:05.150Z
Dave Clarke
<p>I give here my 2 cents, and I add a seventh point:</p>

<ol>
<li>How many reports do I have to do each year?</li>
</ol>

<p><strong>My answers</strong></p>

<ol>
<li><p>Of course the criteria are not the same for conferences and journals. For both correctness is probably the most important criterion. Then, my second criterion is novelty of the results for conferences, while it is impact of the results for journals. For a conference, it is also important to judge whether the paper is of interest for the audience, but this is a judgement call that can be done by PC chairs rather than reviewers.</p></li>
<li><p>First of all, except if you suspect a fraud or bad practice and are not sure (then you should talk solely to the editor), my opinion is that you should tell everything to the authors. I think it is important to be honest with the authors and thus to always give a recommendation. In a report, you should describe the topic, the results (so that authors can see that you understand the paper), then you should assess the merits of the results (novelty w.r.t. the state of the art, technical deepness, correctness, clarity) and point out every problem. Finally, you should give your recommendation, possibly with advice to the authors (such as "I think that this result is nice, but is more fitted for conference A rather than this one"). In a very few cases, you will be given papers that does not deserve to be even submitted, it is not admissible so it must be stated, politely but clearly.</p></li>
<li><p>I think that, except in the case of a journal invitation for a special issue related to a conference paper, there should be significant difference. </p></li>
<li><p>two cases: it is not understandable (their fault) so you can reject the paper for that reason or you are not the good reviewer for that paper. When it happens you know what case it is, if not you are in the second case!</p></li>
<li><p>I am not a native english speaker, so there is a bias in my answer. But I think it is not a reason for rejecting a paper (except in some extreme cases).</p></li>
<li><p>The time needed. It can range from 1 hour (a bad paper in a workshop) to several days (a very involved paper in a top journal).</p></li>
<li><p>Personally, I expect to do 3 reports per paper I submitted (and yes, it's a lot, in fact reviewing probably one of my most time consuming task - before teaching maybe).</p></li>
</ol>

CC BY-SA 2.5
2010-10-03T18:00:03.677Z
Thanks for your input!  Both your answer to 4 and your question 7 raise a good question (which I might make a new #7): when is it appropriate to refuse a refereeing request?
RJK
CC BY-SA 2.5
2010-10-03T20:52:36.967Z
I'd say it is appropriate to refuse a refereeing request when either (1) you do not feel that you have sufficient expertise to do a good review, or (2) you have too many reviewing (or other) commitments at the moment. It is also considered good form to suggest a number of possible alternative reviewers, in the case that you refuse.
Dave Clarke
CC BY-SA 2.5
2010-10-05T15:48:20.733Z
Expecting to do 3 reports per paper submitted matches well with the idea that each paper submitted may get 3 reviews.
Joshua Grochow
CC BY-SA 2.5
2010-10-05T17:51:43.270Z
@Joshua: But then you should divide by the number of authors on the submitted papers. Also I think that very junior people probably submit more than they review, so more senior people have to review more than this calculation would suggest.
Matthias
CC BY-SA 2.5
2010-10-03T17:14:47.003Z
2010-10-03T19:00:42.533Z
2010-10-03T19:00:42.533Z
Sylvain Peyronnet
<p><strong>Updated below</strong></p>

<p>We all know the critical importance of peer-review.  It is the main form of quality control and feedback on research.  However, to an early-stage researcher (like me), it can sometimes be a confusing system/process.</p>

<p>Accordingly, there are several treatises on the scientific refereeing process that give guidance.  Two (very different) examples from computer science -- <a href="http://www.eng.unt.edu/ian/pubs/referee.pdf" rel="noreferrer">this 1994 article by Parberry</a> and <a href="http://portal.acm.org/citation.cfm?doid=1519103.1519122" rel="noreferrer">a more recent one by Cormode</a> -- offer great advice (though the latter might be a shade mischievous).</p>

<p>Here, I'd like to solicit broader advice from the more experienced members of this community about the review process, with particular regard to the peculiarities of theoretical computer science.</p>

<ol>
<li>What are the main criteria for determining the significance of a paper's results?  How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?</li>
<li>What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?</li>
<li>How does assessment for conferences differ from that in journals?  How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?</li>
<li>What if I don't understand the paper? ...the proof?  (Is it my fault or theirs?)</li>
<li>What about typographical/grammatical mistakes? What if there are a lot of them?</li>
<li>How much time should I spend on a report?</li>
<li>How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?</li>
</ol>

<p>Of course, any other relevant questions and answers on this topic are encouraged, since this is CW.</p>

<p>This question is inspired by (stolen from) <a href="https://mathoverflow.net/questions/36596/refereeing-a-paper">a similar post at MathOverflow</a>.</p>

<p><strong>Update 15/02/2011:</strong></p>

<p>I am still very interested in getting more input on this question especially with regard to reviewing conference papers and program committee membership. (These two roles are themselves different beasts, and both very unlike being a referee for a journal article, IMO.)  Granted, program committee membership is rarer than refereeing or reviewing (and it hasn't been my privilege yet), but is a responsibility that every researcher in theoretical computer science must take on eventually.</p>

<ul>
<li><p>Time.  How much time am I expected to spend as a committee member or conference reviewer?  Given the probability that I could get ten or perhaps many more to handle in the space of a few weeks, how do I avoid running out of time?  What are the most important things to spend time on?</p></li>
<li><p>Confidence.  What if the paper is too far from my area of expertise?  What factors should go into nominating/asking someone else to review a submission?  If it is not too far from my area of expertise and I elect to review it, when is it permissible to give a confidence rating of 1?</p></li>
<li><p>Criteria.  There are critical differences between journals and conferences.  Some very important papers are not published in journals.  Some very important papers did not previously appear in conferences.  What are the most significant distinctions in criteria on which to assess papers in these settings?</p></li>
<li><p>Recommendations.  Inherently, there are fewer recommendations that can be offered to the authors of a conference paper, primarily due to space and time constraints.  Also, there is usually only one round of review.  Another consideration is that my report becomes public to the entire strong committee.  What is the scope of suggestions/directives that I can offer?</p></li>
</ul>

<p>As before, if you think I've missed out on asking any particular questions, do let me know, or edit directly.  This is CW, after all.</p>

<p>These new thoughts are partly motivated by reading a paper that <a href="http://geomblog.blogspot.com/2011/02/on-future-of-conferences.html" rel="noreferrer">Suresh mentioned on his blog</a>.</p>

CC BY-SA 2.5
2010-10-03T15:36:04.003Z
Nice question. Since math and CS papers are different in many respects, I think this question will get a lot of interesting responses that differ from the question on MO.
Robin Kothari
CC BY-SA 2.5
2010-10-05T12:18:46.030Z
All three of the current answers have different takes on #3 -- Jeff says most times conference/journal versions are not required to be different; Sylvain suggests that they *are* except for special issues; Dave suggests that many journals insist on a 30% difference. Anybody want to take up this particular point and clarify things?  (I am aware that this must depend a lot on the sub-field.)
RJK
CC BY-SA 2.5
2010-10-06T22:51:49.343Z
Nice question. I wonder why we don't receive the public comments from other reviewers after we referee a paper. That would make it much easier to learn how to referee well. I've seen it done in non-TCS areas of CS, but not in TCS.
Guilherme D. da Fonseca
CC BY-SA 2.5
2010-10-14T21:03:23.310Z
I once wrote a blog post about my thoughts on #7.  Thought I'd share: http://levreyzin.blogspot.com/2010/07/paying-it-backward.html
Lev Reyzin
CC BY-SA 2.5
2010-11-04T09:52:29.910Z
Simone Santini reflected in 2005 on how some classic papers might have fared with modern referees.  Salutary reading: http://arantxa.ii.uam.es/~ssantini/work/papers/recent/s539_sorry_final.pdf or http://www.fang.ece.ufl.edu/reject.html for HTML version.
András Salamon
<soft-question><big-list><paper-review>
How do I referee a paper?
RJK
# How do I referee a paper?

(Asked by: username_0 on 2010-10-03)

**Updated below**

We all know the critical importance of peer-review. It is the main form of quality control and feedback on research. However, to an early-stage researcher (like me), it can sometimes be a confusing system/process.

Accordingly, there are several treatises on the scientific refereeing process that give guidance. Two (very different) examples from computer science -- [this 1994 article by Parberry](http://www.eng.unt.edu/ian/pubs/referee.pdf) and [a more recent one by Cormode](http://portal.acm.org/citation.cfm?doid=1519103.1519122) -- offer great advice (though the latter might be a shade mischievous).

Here, I'd like to solicit broader advice from the more experienced members of this community about the review process, with particular regard to the peculiarities of theoretical computer science.

1. What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?
2. What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?
3. How does assessment for conferences differ from that in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?
4. What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)
5. What about typographical/grammatical mistakes? What if there are a lot of them?
6. How much time should I spend on a report?
7. How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?

Of course, any other relevant questions and answers on this topic are encouraged, since this is CW.

This question is inspired by (stolen from) [a similar post at MathOverflow](https://mathoverflow.net/questions/36596/refereeing-a-paper).

**Update 15/02/2011:**

I am still very interested in getting more input on this question especially with regard to reviewing conference papers and program committee membership. (These two roles are themselves different beasts, and both very unlike being a referee for a journal article, IMO.) Granted, program committee membership is rarer than refereeing or reviewing (and it hasn't been my privilege yet), but is a responsibility that every researcher in theoretical computer science must take on eventually.

* Time. How much time am I expected to spend as a committee member or conference reviewer? Given the probability that I could get ten or perhaps many more to handle in the space of a few weeks, how do I avoid running out of time? What are the most important things to spend time on?
* Confidence. What if the paper is too far from my area of expertise? What factors should go into nominating/asking someone else to review a submission? If it is not too far from my area of expertise and I elect to review it, when is it permissible to give a confidence rating of 1?
* Criteria. There are critical differences between journals and conferences. Some very important papers are not published in journals. Some very important papers did not previously appear in conferences. What are the most significant distinctions in criteria on which to assess papers in these settings?
* Recommendations. Inherently, there are fewer recommendations that can be offered to the authors of a conference paper, primarily due to space and time constraints. Also, there is usually only one round of review. Another consideration is that my report becomes public to the entire strong committee. What is the scope of suggestions/directives that I can offer?

As before, if you think I've missed out on asking any particular questions, do let me know, or edit directly. This is CW, after all.

These new thoughts are partly motivated by reading a paper that [Suresh mentioned on his blog](http://geomblog.blogspot.com/2011/02/on-future-of-conferences.html).

**Tags:** soft-question, big-list, paper-review

> **Comments:**
> **username_1**: Nice question. Since math and CS papers are different in many respects, I think this question will get a lot of interesting responses that differ from the question on MO.  
> **username_0**: All three of the current answers have different takes on #3 -- Jeff says most times conference/journal versions are not required to be different; Sylvain suggests that they *are* except for special issues; Dave suggests that many journals insist on a 30% difference. Anybody want to take up this particular point and clarify things?  (I am aware that this must depend a lot on the sub-field.)  
> **username_2**: Nice question. I wonder why we don't receive the public comments from other reviewers after we referee a paper. That would make it much easier to learn how to referee well. I've seen it done in non-TCS areas of CS, but not in TCS.  
> **username_3**: I once wrote a blog post about my thoughts on #7.  Thought I'd share: http://levreyzin.blogspot.com/2010/07/paying-it-backward.html  
> **username_4**: Simone Santini reflected in 2005 on how some classic papers might have fared with modern referees.  Salutary reading: http://arantxa.ii.uam.es/~ssantini/work/papers/recent/s539_sorry_final.pdf or http://www.fang.ece.ufl.edu/reject.html for HTML version.  

---

## 3 answers

### Answer by username_5 on 2010-10-03. Score: 61

1. To the best of your knowledge, does the paper make a significant, well-presented, and correct contribution to the state of the art? If the paper fails any of the three criteria, it's fair to reject it for that reason alone, regardless of the other two.
2. Here's what I think a report should contain. Everything should be visible to the author, except *possibly* for serious accusations of misconduct.

   a. A quick summary of the paper, to help the editor judge the quality of the results, and to help convince both the author and the editor that you actually read and understood the paper. Place the result in its larger context. Include a history of prior versions, even if the authors include it in the submission. Be respectful, but brutally honest.

   b. A discussion of the strengths and weaknesses of the paper, in terms of correctness, novelty, clarity, importance, generality, potential impact, elegance, technical depth, robustness, etc. If you suspect unethical behavior (plagiarism, parallel submission, cooked data), describe your suspicions. Be respectful, but brutally honest.

   c. A recommendation to the editor for further action — accept, accept with minor revision, ask for a second round of reviewing, or reject outright. Keep in mind that you are making a recommendation, not a decision; if you can't make up your mind, just say so. Be respectful, but brutally honest.

   d. More detailed feedback to the author — more detailed justification for your recommendation, requests for clarification in the final version, missing references, bugs in the proofs, simplifications, generalizations, typos, etc. Be respectful, but brutally honest.
3. Conference reports should be shorter; program committees have hundreds of papers to consider at once. Whether there should be a difference between conference and journal papers is up to the journal (and indirectly, up to the community). Most theoretical computer science journals do *not* insist on a significant difference; it is quite common for the conference and journal versions of a theory paper to be essentially identical. When in doubt, ask the editor!
4. If you still don't understand the paper after making a good-faith effort, it's the author's fault, or possibly the editor's, but *certainly not* yours. The author's primary responsibility is to effectively communicate their result to their audience, and a good editor will send you a paper to referee only if they think you're a good representative of the paper's intended audience. But you do have to make a good-faith effort; do not expect to immediately understand everything (anything?) immediately on your first reading.
5. If there are a *lot* of errors, don't even read the paper; just recommend rejection on the grounds that the paper is not professionally written. Otherwise, if you really want to be thorough, include a *representative* list of grammar, spelling, and punctuation mistakes, but don't knock yourself out finding every last bug. Be respectful, but brutally honest.
6. Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques. Be pleasantly surprised when it doesn't actually take that long. (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. Don't confuse these three possibilities.)
7. You should write at least as many referee reports as other people write for you. If this takes more time than writing your own papers, you're not spending enough time on your own papers.

> **Comments:**
> **username_0**: Nice punchy summary! I especially like 2a-d.  
> **username_6**: My favorite part is "If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. **Don't confuse these three possibilities.**"  

---

### Answer by username_7 on 2010-10-03. Score: 21

A lot depends upon the conference/journal, as each community has developed its own style, so knowing what's expected of the conference/journal certainly will help a little.

> **1.** What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?

Criteria: Novelty/originality, expected impact, correctness/validity, extensiveness (how much of the problem is studied? + theorems? + implementation? +experimental results?), quality of presentation. The criteria for journal acceptance are much higher than for conferences. For a conference you give a score based on the previous criteria. Verifying correctness as much as is possible is very important, especially for journal articles.

> **2.** What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?

The referees report contains at least 4 main elements: A summary of the paper and its contributions; points in favour of accepting the paper; points against the paper; major comments, including points to be addressed; minor comments (typos etc). You should always give an indication of acceptance or rejection or the degree of revision required. Comments to the editor could include: a short, perhaps blunt assessment of the paper; any statement of doubt; details of possible plagiarism or parallel submission of the paper; ...

> **3.** How does assessment for conferences differ from those in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?

Journal reviews tend to be more extensive and can include a lot of things for the authors
to do to bring their paper into an acceptable state, such as "implement and evaluate these ideas". Your confidence should be based on how confident you feel. Generally this will come with experience, so start out being conservative. It it's out of your field, even though you may understand the paper, being conservative is also a good idea –– it's difficult to assess originality of a paper outside of your field. Some journals say that journal submissions must contain a substantial amount of new material compared to the conference version. 30% is a figure I've heard.

> **4.** What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)

This can vary. Sometimes it will be your fault, sometimes it will be theirs. Use your judgement. Maybe ask a colleague to take a look at the paper. If you totally cannot understand it and it's not due to poor formatting, then perhaps contact the PC chair and explain that this is the case. In any case, this should be reflected in your confidence score. If the paper cannot be understood due to poor writing or language or because it has been prematurely submitted, then this should be written in your report.

> **5.** What about typographical/grammatical mistakes? What if there are a lot of them?

When I was younger I'd report every single typo and grammatical error. Now I just don't have the time. Reporting some is always helpful: pick the most serious ones. Also recommend that the paper be proofread (by a native speaker). For journal papers be more thorough. If it has way too many errors, then is shouldn't have been submitted to a journal, which alone is reason for rejection (IMHO).

> **6.** How much time should I spend on a report?

One day maximum for a conference paper, including reading time. For journal papers, especially long ones, reading the paper carefully might take a whole week.

> **7.** How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?

The expected amount could vary between 5 and 30. As a junior researcher, you'll be given a few for practice. As you advance through the grades, you'll take on more reviewing as you become a member of PCs. Then as you advance even further, you'll have an army of PhD students to do your reviewing for you. More than 30 in a year is quite onerous.

As I said above, I'd say it is appropriate to refuse a refereeing request when either (1) you do not feel that you have sufficient expertise to do a good review, or (2) you have too many reviewing (or other) commitments at the moment. It is also considered good form to suggest a number of possible alternative reviewers, in the case that you refuse.


---

### Answer by username_8 on 2010-10-03. Score: 13

I give here my 2 cents, and I add a seventh point:

1. How many reports do I have to do each year?

**My answers**

1. Of course the criteria are not the same for conferences and journals. For both correctness is probably the most important criterion. Then, my second criterion is novelty of the results for conferences, while it is impact of the results for journals. For a conference, it is also important to judge whether the paper is of interest for the audience, but this is a judgement call that can be done by PC chairs rather than reviewers.
2. First of all, except if you suspect a fraud or bad practice and are not sure (then you should talk solely to the editor), my opinion is that you should tell everything to the authors. I think it is important to be honest with the authors and thus to always give a recommendation. In a report, you should describe the topic, the results (so that authors can see that you understand the paper), then you should assess the merits of the results (novelty w.r.t. the state of the art, technical deepness, correctness, clarity) and point out every problem. Finally, you should give your recommendation, possibly with advice to the authors (such as "I think that this result is nice, but is more fitted for conference A rather than this one"). In a very few cases, you will be given papers that does not deserve to be even submitted, it is not admissible so it must be stated, politely but clearly.
3. I think that, except in the case of a journal invitation for a special issue related to a conference paper, there should be significant difference.
4. two cases: it is not understandable (their fault) so you can reject the paper for that reason or you are not the good reviewer for that paper. When it happens you know what case it is, if not you are in the second case!
5. I am not a native english speaker, so there is a bias in my answer. But I think it is not a reason for rejecting a paper (except in some extreme cases).
6. The time needed. It can range from 1 hour (a bad paper in a workshop) to several days (a very involved paper in a top journal).
7. Personally, I expect to do 3 reports per paper I submitted (and yes, it's a lot, in fact reviewing probably one of my most time consuming task - before teaching maybe).

> **Comments:**
> **username_0**: Thanks for your input!  Both your answer to 4 and your question 7 raise a good question (which I might make a new #7): when is it appropriate to refuse a refereeing request?  
> **username_7**: I'd say it is appropriate to refuse a refereeing request when either (1) you do not feel that you have sufficient expertise to do a good review, or (2) you have too many reviewing (or other) commitments at the moment. It is also considered good form to suggest a number of possible alternative reviewers, in the case that you refuse.  
> **username_9**: Expecting to do 3 reports per paper submitted matches well with the idea that each paper submitted may get 3 reviews.  
> **username_10**: @Joshua: But then you should divide by the number of authors on the submitted papers. Also I think that very junior people probably submit more than they review, so more senior people have to review more than this calculation would suggest.  

---

