<p>I get the impression that good journal editors will get at least one reviewer who is skilled in the methodology used in the paper. The importance of this reviewer role would presumably vary with the statistical or other methodological complexity of the paper. </p>

<p>That said, reviewing is well known to be imperfect particularly when it comes to checking all the possible smaller errors that can be made.  Errors in statistical analysis and reporting are wide-spread in published articles in psychology. For example, Bakker and Wicherts (2011) did a review, where to quote the abstract </p>

<blockquote>
  <p>we checked the consistency of reported
  test statistics, degrees of freedom, and p values in a random sample
  of high- and low-impact psychology journals... On the basis of 281
  articles ... [we estimate that] around 18% of statistical results in the
  psychological literature are incorrectly reported. Inconsistencies
  were more common in low-impact journals than in high-impact journals.
  Moreover, around 15% of the articles contained at least one
  statistical conclusion that proved, upon recalculation, to be
  incorrect; that is, recalculation rendered the previously significant
  result insignificant, or vice versa. </p>
</blockquote>

<h3>References</h3>

<ul>
<li>Bakker, M., &amp; Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678. <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3174372/" rel="nofollow">FULL-TEXT</a></li>
</ul>

CC BY-SA 3.0
2013-08-22T10:49:00.963Z
2013-08-22T10:49:00.963Z
Jeromy Anglim
<p>Usually an article goes through a peer-review process before it is published, and from what I here it is quite common that the reviewers "demand" some changes.</p>

<ul>
<li>Are there guidelines for peer- reviewers, especially as the methodological side of things is concerned?</li>
<li>Are  "methodology specialists" incorporated in the process? </li>
</ul>

<methodology><publication-process>
Are there guidelines in the peer reviewing process on assessing methodology?
Jens Kouros
# Are there guidelines in the peer reviewing process on assessing methodology?

(Asked by: username_0 on 2013-08-21)

Usually an article goes through a peer-review process before it is published, and from what I here it is quite common that the reviewers "demand" some changes.

* Are there guidelines for peer- reviewers, especially as the methodological side of things is concerned?
* Are  "methodology specialists" incorporated in the process?

**Tags:** methodology, publication-process


---

## 1 answers

### Answer by username_1 on 2013-08-22. Score: 3 | Accepted answer

I get the impression that good journal editors will get at least one reviewer who is skilled in the methodology used in the paper. The importance of this reviewer role would presumably vary with the statistical or other methodological complexity of the paper.

That said, reviewing is well known to be imperfect particularly when it comes to checking all the possible smaller errors that can be made. Errors in statistical analysis and reporting are wide-spread in published articles in psychology. For example, Bakker and Wicherts (2011) did a review, where to quote the abstract

> we checked the consistency of reported
> test statistics, degrees of freedom, and p values in a random sample
> of high- and low-impact psychology journals... On the basis of 281
> articles ... [we estimate that] around 18% of statistical results in the
> psychological literature are incorrectly reported. Inconsistencies
> were more common in low-impact journals than in high-impact journals.
> Moreover, around 15% of the articles contained at least one
> statistical conclusion that proved, upon recalculation, to be
> incorrect; that is, recalculation rendered the previously significant
> result insignificant, or vice versa.

### References

* Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678. [FULL-TEXT](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3174372/)


---

