  In this paper peer review reliability is investigated based on peer ratings
of research teams at two Belgian universities. It is found that outcomes can be
substantially influenced by the different ways in which experts attribute
ratings. To increase reliability of peer ratings, procedures creating a uniform
reference level should be envisaged. One should at least check for signs of low
reliability, which can be obtained from an analysis of the outcomes of the peer
evaluation itself. The peer review results are compared to outcomes from a
citation analysis of publications by the same teams, in subject fields well
covered by citation indexes. It is illustrated how, besides reliability,
comparability of results depends on the nature of the indicators, on the
subject area and on the intrinsic characteristics of the methods. The results
further confirm what is currently considered as good practice: the presentation
of results for not one but for a series of indicators.
