  Due to the massive size of test collections, a standard practice in IR
evaluation is to construct a 'pool' of candidate relevant documents comprised
of the top-k documents retrieved by a wide range of different retrieval systems
- a process called depth-k pooling. A standard practice is to set the depth (k)
to a constant value for each query constituting the benchmark set. However, in
this paper we argue that the annotation effort can be substantially reduced if
the depth of the pool is made a variable quantity for each query, the rationale
being that the number of documents relevant to the information need can widely
vary across queries. Our hypothesis is that a lower depth for the former class
of queries and a higher depth for the latter can potentially reduce the
annotation effort without a significant change in retrieval effectiveness
evaluation. We make use of standard query performance prediction (QPP)
techniques to estimate the number of potentially relevant documents for each
query, which is then used to determine the depth of the pool. Our experiments
conducted on standard test collections demonstrate that this proposed method of
employing query-specific variable depths is able to adequately reflect the
relative effectiveness of IR systems with a substantially smaller annotation
effort.
