  As an important biomedical database, PubMed provides users with free access
to abstracts of its documents. However, citations between these documents need
to be collected from external data sources. Although previous studies have
investigated the coverage of various data sources, the quality of citations is
underexplored. In response, this study compares the coverage and citation
quality of five freely available data sources on 30 million PubMed documents,
including OpenCitations Index of CrossRef open DOI-to-DOI citations (COCI),
Dimensions, Microsoft Academic Graph (MAG), National Institutes of Health Open
Citation Collection (NIH-OCC), and Semantic Scholar Open Research Corpus
(S2ORC). Three gold standards and five metrics are introduced to evaluate the
correctness and completeness of citations. Our results indicate that Dimensions
is the most comprehensive data source that provides references for 62.4% of
PubMed documents, outperforming the official NIH-OCC dataset (56.7%). Over 90%
of citation links in other data sources can also be found in Dimensions. The
coverage of MAG, COCI, and S2ORC is 59.6%, 34.7%, and 23.5%, respectively.
Regarding the citation quality, Dimensions and NIH-OCC achieve the best overall
results. Almost all data sources have a precision higher than 90%, but their
recall is much lower. All databases have better performances on recent
publications than earlier ones. Meanwhile, the gaps between different data
sources have diminished for the documents published in recent years. This study
provides evidence for researchers to choose suitable PubMed citation sources,
which is also helpful for evaluating the citation quality of free bibliographic
databases.
