  Research community evaluations in information retrieval, such as NIST's Text
REtrieval Conference (TREC), build reusable test collections by pooling
document rankings submitted by many teams. Naturally, the quality of the
resulting test collection thus greatly depends on the number of participating
teams and the quality of their submitted runs. In this work, we investigate: i)
how the number of participants, coupled with other factors, affects the quality
of a test collection; and ii) whether the quality of a test collection can be
inferred prior to collecting relevance judgments from human assessors.
Experiments conducted on six TREC collections illustrate how the number of
teams interacts with various other factors to influence the resulting quality
of test collections. We also show that the reusability of a test collection can
be predicted with high accuracy when the same document collection is used for
successive years in an evaluation campaign, as is common in TREC.
