  STATCHECK is an R algorithm designed to scan papers automatically for
inconsistencies between test statistics and their associated p values (Nuijten
et al., 2016). The goal of this comment is to point out an important and
well-documented flaw in this busily applied algorithm: It cannot handle
corrected p values. As a result, statistical tests applying appropriate
corrections to the p value (e.g., for multiple tests, post-hoc tests,
violations of assumptions, etc.) are likely to be flagged as reporting
inconsistent statistics, whereas papers omitting necessary corrections are
certified as correct. The STATCHECK algorithm is thus valid for only a subset
of scientific papers, and conclusions about the quality or integrity of
statistical reports should never be based solely on this program.
