  To create a new IR test collection at low cost, it is valuable to carefully
select which documents merit human relevance judgments. Shared task campaigns
such as NIST TREC pool document rankings from many participating systems (and
often interactive runs as well) in order to identify the most likely relevant
documents for human judging. However, if one's primary goal is merely to build
a test collection, it would be useful to be able to do so without needing to
run an entire shared task. Toward this end, we investigate multiple active
learning strategies which, without reliance on system rankings: 1) select which
documents human assessors should judge; and 2) automatically classify the
relevance of additional unjudged documents. To assess our approach, we report
experiments on five TREC collections with varying scarcity of relevant
documents. We report labeling accuracy achieved, as well as rank correlation
when evaluating participant systems based upon these labels vs.\ full pool
judgments. Results show the effectiveness of our approach, and we further
analyze how varying relevance scarcity across collections impacts our findings.
To support reproducibility and follow-on work, we have shared our code online:
https://github.com/mdmustafizurrahman/ICTIR_AL_TestCollection_2020/.
