  There are many different relatedness measures, based for instance on citation
relations or textual similarity, that can be used to cluster scientific
publications. We propose a principled methodology for evaluating the accuracy
of clustering solutions obtained using these relatedness measures. We formally
show that the proposed methodology has an important consistency property. The
empirical analyses that we present are based on publications in the fields of
cell biology, condensed matter physics, and economics. Using the BM25
text-based relatedness measure as evaluation criterion, we find that
bibliographic coupling relations yield more accurate clustering solutions than
direct citation relations and co-citation relations. The so-called extended
direct citation approach performs similarly to or slightly better than
bibliographic coupling in terms of the accuracy of the resulting clustering
solutions. The other way around, using a citation-based relatedness measure as
evaluation criterion, BM25 turns out to yield more accurate clustering
solutions than other text-based relatedness measures.
