  This study explores the extent to which bibliometric indicators based on
counts of highly-cited documents could be affected by the choice of data
source. The initial hypothesis is that databases that rely on journal selection
criteria for their document coverage may not necessarily provide an accurate
representation of highly-cited documents across all subject areas, while
inclusive databases, which give each document the chance to stand on its own
merits, might be better suited to identify highly-cited documents. To test this
hypothesis, an analysis of 2,515 highly-cited documents published in 2006 that
Google Scholar displays in its Classic Papers product is carried out at the
level of broad subject categories, checking whether these documents are also
covered in Web of Science and Scopus, and whether the citation counts offered
by the different sources are similar. The results show that a large fraction of
highly-cited documents in the Social Sciences and Humanities (8.6%-28.2%) are
invisible to Web of Science and Scopus. In the Natural, Life, and Health
Sciences the proportion of missing highly-cited documents in Web of Science and
Scopus is much lower. Furthermore, in all areas, Spearman correlation
coefficients of citation counts in Google Scholar, as compared to Web of
Science and Scopus citation counts, are remarkably strong (.83-.99). The main
conclusion is that the data about highly-cited documents available in the
inclusive database Google Scholar does indeed reveal significant coverage
deficiencies in Web of Science and Scopus in several areas of research.
Therefore, using these selective databases to compute bibliometric indicators
based on counts of highly-cited documents might produce biased assessments in
poorly covered areas.
