  In many contemporary applications, large amounts of unlabeled data are
readily available while labeled examples are limited. There has been
substantial interest in semi-supervised learning (SSL) which aims to leverage
unlabeled data to improve estimation or prediction. However, current SSL
literature focuses primarily on settings where labeled data is selected
randomly from the population of interest. Non-random sampling, while posing
additional analytical challenges, is highly applicable to many real world
problems. Moreover, no SSL methods currently exist for estimating the
prediction performance of a fitted model under non-random sampling. In this
paper, we propose a two-step SSL procedure for evaluating a prediction rule
derived from a working binary regression model based on the Brier score and
overall misclassification rate under stratified sampling. In step I, we impute
the missing labels via weighted regression with nonlinear basis functions to
account for nonrandom sampling and to improve efficiency. In step II, we
augment the initial imputations to ensure the consistency of the resulting
estimators regardless of the specification of the prediction model or the
imputation model. The final estimator is then obtained with the augmented
imputations. We provide asymptotic theory and numerical studies illustrating
that our proposals outperform their supervised counterparts in terms of
efficiency gain. Our methods are motivated by electronic health records (EHR)
research and validated with a real data analysis of an EHR-based study of
diabetic neuropathy.
