  Little is known whether peer reviewers use the same evaluation criteria and
how they integrate the criteria into their overall judgment. This study
therefore proposed two assessment styles based on theoretical perspectives and
normative positions. According to the case-by-case style, referees use many and
different criteria, weight criteria on a case-by-case basis, and integrate
criteria in a complex, non-mechanical way into their overall judgment.
According to the uniform style, referees use a small fraction of the available
criteria, apply the same criteria, weight the criteria in the same way, and
integrate the criteria based on simple rules (i.e., fast-and-frugal
heuristics). These two styles were examined using a unique dataset from a
career funding scheme that contained a comparatively large number of evaluation
criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic
regression) were employed to describe how referees integrate the criteria into
their overall judgment. The logistic regression predicted the referees' overall
assessment with high accuracy and slightly more accurately than the
fast-and-frugal trees. Overall, the results of this study support the uniform
style but also indicate that the uniform style needs to be revised as follows:
referees use many criteria and integrate the criteria using complex rules.
However, and most importantly, the revised style could describe most - but not
all - of the referees' judgments. Future studies should therefore examine how
referees' judgments can be characterized in those cases where the uniform style
failed. Moreover, the evaluation process of referees should be studied in more
empirical and theoretical detail.
