  This study examines a basic assumption of peer review, namely, the idea that
there is a consensus on evaluation criteria among peers, which is a necessary
condition for the reliability of peer judgements. Empirical evidence indicating
that there is no consensus or more than one consensus would offer an
explanation for the disagreement effect, the low inter-rater reliability
consistently observed in peer review. To investigate this basic assumption, we
have surveyed all humanities scholars in Switzerland on 23 grant review
criteria. We have employed latent class tree modelling to identify subgroups in
which scholars rated criteria similarly (i.e. latent classes) and to explore
covariates predicting class membership. We have identified two consensus
classes, two consensus-close classes, and a consensus-far class. The consensus
classes contain a core consensus (ten criteria related to knowledge gaps,
feasibility, rigour, comprehensibility and argumentation, and academic
relevance, as well as to the competence and experience of the applicant) and a
broad consensus that includes the core consensus plus eight
contribution-related criteria, such as originality. These results provide a
possible explanation for the disagreement effect. Moreover, the results are
consistent with the notion of conservatism, which holds that original research
is undervalued in peer review, while other aspects, such as methodology and
feasibility, are overweighted. The covariate analysis indicated that age and
having tenure increases from the consensus-far to the consensus-close to the
consensus classes. This suggests that the more academic experience scholars
accumulate, the more their understanding of review criteria conforms to the
social norm.
