  Retrieval models are often evaluated on partially-annotated datasets. Each
query is mapped to a few relevant texts and the remaining corpus is assumed to
be irrelevant. As a result, models that successfully retrieve false negatives
are punished in evaluation. Unfortunately, completely annotating all texts for
every query is not resource efficient. In this work, we show that using
partially-annotated datasets in evaluation can paint a distorted picture. We
curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
contain all relevant passages for each query. Queries describe a group (e.g.,
``journals about linguistics'') and relevant passages are evidence that
entities belong to the group (e.g., a passage indicating that Language is a
journal about linguistics). We show that evaluating on a dataset containing
annotations for only a subset of the relevant passages might result in
misleading ranking of the retrieval systems and that as more relevant texts are
included in the evaluation set, the rankings converge. We propose our dataset
as a resource for evaluation and our study as a recommendation for balance
between resource-efficiency and reliable evaluation when annotating evaluation
sets for text retrieval.
