  Receiver operating characteristic (ROC) analysis is widely used for
evaluating diagnostic systems. Recent studies have shown that estimating an
area under ROC curve (AUC) with standard cross-validation methods suffers from
a large bias. The leave-pair-out (LPO) cross-validation has been shown to
correct this bias. However, while LPO produces an almost unbiased estimate of
AUC, it does not provide a ranking of the data needed for plotting and
analyzing the ROC curve. In this study, we propose a new method called
tournament leave-pair-out (TLPO) cross-validation. This method extends LPO by
creating a tournament from pair comparisons to produce a ranking for the data.
TLPO preserves the advantage of LPO for estimating AUC, while it also allows
performing ROC analyses. We have shown using both synthetic and real world data
that TLPO is as reliable as LPO for AUC estimation, and confirmed the bias in
leave-one-out cross-validation on low-dimensional data. As a case study on ROC
analysis, we also evaluate how reliably sensitivity and specificity can be
estimated from TLPO ROC curves.
