  Wikidata is one of the most important sources of structured data on the web,
built by a worldwide community of volunteers. As a secondary source, its
contents must be backed by credible references; this is particularly important
as Wikidata explicitly encourages editors to add claims for which there is no
broad consensus, as long as they are corroborated by references. Nevertheless,
despite this essential link between content and references, Wikidata's ability
to systematically assess and assure the quality of its references remains
limited. To this end, we carry out a mixed-methods study to determine the
relevance, ease of access, and authoritativeness of Wikidata references, at
scale and in different languages, using online crowdsourcing, descriptive
statistics, and machine learning. Building on previous work of ours, we run a
series of microtasks experiments to evaluate a large corpus of references,
sampled from Wikidata triples with labels in several languages. We use a
consolidated, curated version of the crowdsourced assessments to train several
machine learning models to scale up the analysis to the whole of Wikidata. The
findings help us ascertain the quality of references in Wikidata, and identify
common challenges in defining and capturing the quality of user-generated
multilingual structured data on the web. We also discuss ongoing editorial
practices, which could encourage the use of higher-quality references in a more
immediate way. All data and code used in the study are available on GitHub for
feedback and further improvement and deployment by the research community.
