  Neural retrieval models are generally regarded as fundamentally different
from the retrieval techniques used in the late 1990's when the TREC ad hoc test
collections were constructed. They thus provide the opportunity to empirically
test the claim that pooling-built test collections can reliably evaluate
retrieval systems that did not contribute to the construction of the collection
(in other words, that such collections can be reusable). To test the
reusability claim, we asked TREC assessors to judge new pools created from new
search results for the TREC-8 ad hoc collection. These new search results
consisted of five new runs (one each from three transformer-based models and
two baseline runs that use BM25) plus the set of TREC-8 submissions that did
not previously contribute to pools. The new runs did retrieve previously unseen
documents, but the vast majority of those documents were not relevant. The
ranking of all runs by mean evaluation score when evaluated using the official
TREC-8 relevance judgment set and the newly expanded relevance set are almost
identical, with Kendall's tau correlations greater than 0.99. Correlations for
individual topics are also high. The TREC-8 ad hoc collection was originally
constructed using deep pools over a diverse set of runs, including several
effective manual runs. Its judgment budget, and hence construction cost, was
relatively large. However, it does appear that the expense was well-spent: even
with the advent of neural techniques, the collection has stood the test of time
and remains a reliable evaluation instrument as retrieval techniques have
advanced.
