  Predictive algorithms inform consequential decisions in settings where the
outcome is selectively observed given choices made by human decision makers. We
propose a unified framework for the robust design and evaluation of predictive
algorithms in selectively observed data. We impose general assumptions on how
much the outcome may vary on average between unselected and selected units
conditional on observed covariates and identified nuisance parameters,
formalizing popular empirical strategies for imputing missing data such as
proxy outcomes and instrumental variables. We develop debiased machine learning
estimators for the bounds on a large class of predictive performance estimands,
such as the conditional likelihood of the outcome, a predictive algorithm's
mean square error, true/false positive rate, and many others, under these
assumptions. In an administrative dataset from a large Australian financial
institution, we illustrate how varying assumptions on unobserved confounding
leads to meaningful changes in default risk predictions and evaluations of
credit scores across sensitive groups.
