  Although various citation-based indicators are commonly used to help research
evaluations, there are ongoing controversies about their value. In response,
they are often correlated with quality ratings or with other quantitative
indicators in order to partly assess their validity. When correlations are
calculated for sets of publications from multiple disciplines or years,
however, the magnitude of the correlation coefficient may be reduced, masking
the strength of the underlying correlation. In response, this article uses
simulations to systematically investigate the extent to which mixing years or
disciplines reduces correlations. The results show that mixing two sets of
articles with different correlation strengths can reduce the correlation for
the combined set to substantially below the average of the two. Moreover, even
mixing two sets of articles with the same correlation strength but different
mean citation counts can substantially reduce the correlation for the combined
set. The extent of the reduction in correlation also depends upon whether the
articles assessed have been pre-selected for being high quality and whether the
relationship between the quality ratings and citation counts is linear or
exponential. The results underline the importance of using homogeneous data
sets but also help to interpret correlation coefficients when this is
impossible.
