  Algorithms are commonly used to predict outcomes under a particular decision
or intervention, such as predicting whether an offender will succeed on parole
if placed under minimal supervision. Generally, to learn such counterfactual
prediction models from observational data on historical decisions and
corresponding outcomes, one must measure all factors that jointly affect the
outcomes and the decision taken. Motivated by decision support applications, we
study the counterfactual prediction task in the setting where all relevant
factors are captured in the historical data, but it is either undesirable or
impermissible to use some such factors in the prediction model. We refer to
this setting as runtime confounding. We propose a doubly-robust procedure for
learning counterfactual prediction models in this setting. Our theoretical
analysis and experimental results suggest that our method often outperforms
competing approaches. We also present a validation procedure for evaluating the
performance of counterfactual prediction methods.
