  This paper appraises the concordance between bibliometrics and peer review,
by drawing evidence from the data of two experiments realized by the Italian
governmental agency for research evaluation. The experiments were performed for
validating the dual system of evaluation, consisting in the interchangeable use
of bibliometyrics and peer review, adopted by the agency in the research
assessment exercises. The two experiments were based on stratified random
samples of journal articles. Each article was scored by bibliometrics and by
peer review. The degree of concordance between the two evaluations is then
computed. The correct setting of the experiments is defined by developing the
design-based estimation of the Cohen's kappa coefficient and some testing
procedures for assessing the homogeneity of missing proportions between strata.
The results of both experiments show that for each research areas of hard
sciences, engineering and life sciences, the degree of agreement between
bibliometrics and peer review is -- at most -- weak at an individual article
level. Thus, the outcome of the experiments does not validate the use of the
dual system of evaluation in the Italian research assessments. More in general,
the very weak concordance indicates that metrics should not replace peer review
at the level of individual article. Hence, the use of the dual system of
evaluation for reducing costs might introduce unknown biases in a research
assessment exercise.
