  The MS MARCO-passage dataset has been the main large-scale dataset open to
the IR community and it has fostered successfully the development of novel
neural retrieval models over the years. But, it turns out that two different
corpora of MS MARCO are used in the literature, the official one and a second
one where passages were augmented with titles, mostly due to the introduction
of the Tevatron code base. However, the addition of titles actually leaks
relevance information, while breaking the original guidelines of the MS
MARCO-passage dataset. In this work, we investigate the differences between the
two corpora and demonstrate empirically that they make a significant difference
when evaluating a new method. In other words, we show that if a paper does not
properly report which version is used, reproducing fairly its results is
basically impossible. Furthermore, given the current status of reviewing, where
monitoring state-of-the-art results is of great importance, having two
different versions of a dataset is a large problem. This is why this paper aims
to report the importance of this issue so that researchers can be made aware of
this problem and appropriately report their results.
