  While test collections provide the cornerstone for Cranfield-based evaluation
of information retrieval (IR) systems, it has become practically infeasible to
rely on traditional pooling techniques to construct test collections at the
scale of today's massive document collections. In this paper, we propose a new
intelligent topic selection method which reduces the number of search topics
needed for reliable IR evaluation. To rigorously assess our method, we
integrate previously disparate lines of research on intelligent topic selection
and deep vs. shallow judging. While prior work on intelligent topic selection
has never been evaluated against shallow judging baselines, prior work on deep
vs. shallow judging has largely argued for shallowed judging, but assuming
random topic selection. We argue that for evaluating any topic selection
method, ultimately one must ask whether it is actually useful to select topics,
or should one simply perform shallow judging over many topics? In seeking a
rigorous answer to this over-arching question, we conduct a comprehensive
investigation over a set of relevant factors never previously studied together
1) topic selection method 2) the effect of topic familiarity on human judging
speed and 3) how different topic generation processes impact (i) budget
utilization and (ii) the resultant quality of judgments. Experiments on NIST
TREC Robust 2003 and Robust 2004 test collections show that not only can we
reliably evaluate IR systems with fewer topics, but also that 1) when topics
are intelligently selected, deep judging is often more cost-effective than
shallow judging in evaluation reliability and 2) topic familiarity and topic
generation costs greatly impact the evaluation cost vs. reliability trade-off.
Our findings challenge conventional wisdom in showing that deep judging is
often preferable to shallow judging when topics are selected intelligently.
