  Survivorship bias is the tendency to concentrate on the positive outcomes of
a selection process and overlook the results that generate negative outcomes.
We observe that this bias could be present in the popular MS MARCO dataset,
given that annotators could not find answers to 38--45% of the queries, leading
to these queries being discarded in training and evaluation processes. Although
we find that some discarded queries in MS MARCO are ill-defined or otherwise
unanswerable, many are valid questions that could be answered had the
collection been annotated more completely (around two thirds using modern
ranking techniques). This survivability problem distorts the MS MARCO
collection in several ways. We find that it affects the natural distribution of
queries in terms of the type of information needed. When used for evaluation,
we find that the bias likely yields a significant distortion of the absolute
performance scores observed. Finally, given that MS MARCO is frequently used
for model training, we train models based on subsets of MS MARCO that simulates
more survivorship bias. We find that models trained in this setting are up to
9.9% worse when evaluated on versions of the dataset with more complete
annotations, and up to 3.5% worse at zero-shot transfer. Our findings are
complementary to other recent suggestions for further annotation of MS MARCO,
but with a focus on discarded queries. Code and data for reproducing the
results of this paper are available in an online appendix.
