  Because researchers typically do not have the time or space to present more
than a few evaluation metrics in any published study, it can be difficult to
assess relative effectiveness of prior methods for unreported metrics when
baselining a new method or conducting a systematic meta-review. While sharing
of study data would help alleviate this, recent attempts to encourage
consistent sharing have been largely unsuccessful. Instead, we propose to
enable relative comparisons with prior work across arbitrary metrics by
predicting unreported metrics given one or more reported metrics. In addition,
we further investigate prediction of high-cost evaluation measures using
low-cost measures as a potential strategy for reducing evaluation cost. We
begin by assessing the correlation between 23 IR metrics using 8 TREC test
collections. Measuring prediction error wrt. R-square and Kendall's tau, we
show that accurate prediction of MAP, P@10, and RBP can be achieved using only
2-3 other metrics. With regard to lowering evaluation cost, we show that
RBP(p=0.95) can be predicted with high accuracy using measures with only
evaluation depth of 30. Taken together, our findings provide a valuable
proof-of-concept which we expect to spur follow-on work by others in proposing
more sophisticated models for metric prediction.
