  The recent MSMARCO passage retrieval collection has allowed researchers to
develop highly tuned retrieval systems. One aspect of this data set that makes
it distinctive compared to traditional corpora is that most of the topics only
have a single answer passage marked relevant. Here we carry out a "what if"
sensitivity study, asking whether a set of systems would still have the same
relative performance if more passages per topic were deemed to be "relevant",
exploring several mechanisms for identifying sets of passages to be so
categorized. Our results show that, in general, while run scores can vary
markedly if additional plausible passages are presumed to be relevant, the
derived system ordering is relatively insensitive to additional relevance,
providing support for the methodology that was used at the time the MSMARCO
passage collection was created.
