  In many modern machine learning applications, the outcome is expensive or
time-consuming to collect while the predictor information is easy to obtain.
Semi-supervised learning (SSL) aims at utilizing large amounts of `unlabeled'
data along with small amounts of `labeled' data to improve the efficiency of a
classical supervised approach. Though numerous SSL classification and
prediction procedures have been proposed in recent years, no methods currently
exist to evaluate the prediction performance of a working regression model. In
the context of developing phenotyping algorithms derived from electronic
medical records (EMR), we present an efficient two-step estimation procedure
for evaluating a binary classifier based on various prediction performance
measures in the semi-supervised (SS) setting. In step I, the labeled data is
used to obtain a non-parametrically calibrated estimate of the conditional risk
function. In step II, SS estimates of the prediction accuracy parameters are
constructed based on the estimated conditional risk function and the unlabeled
data. We demonstrate that under mild regularity conditions the proposed
estimators are consistent and asymptotically normal. Importantly, the
asymptotic variance of the SS estimators is always smaller than that of the
supervised counterparts under correct model specification. We also correct for
potential overfitting bias in the SS estimators in finite sample with
cross-validation and develop a perturbation resampling procedure to approximate
their distributions. Our proposals are evaluated through extensive simulation
studies and illustrated with two real EMR studies aiming to develop phenotyping
algorithms for rheumatoid arthritis and multiple sclerosis.
