  Machine learning can help personalized decision support by learning models to
predict individual treatment effects (ITE). This work studies the reliability
of prediction-based decision-making in a task of deciding which action $a$ to
take for a target unit after observing its covariates $\tilde{x}$ and predicted
outcomes $\hat{p}(\tilde{y} \mid \tilde{x}, a)$. An example case is
personalized medicine and the decision of which treatment to give to a patient.
A common problem when learning these models from observational data is
imbalance, that is, difference in treated/control covariate distributions,
which is known to increase the upper bound of the expected ITE estimation
error. We propose to assess the decision-making reliability by estimating the
ITE model's Type S error rate, which is the probability of the model inferring
the sign of the treatment effect wrong. Furthermore, we use the estimated
reliability as a criterion for active learning, in order to collect new
(possibly expensive) observations, instead of making a forced choice based on
unreliable predictions. We demonstrate the effectiveness of this
decision-making aware active learning in two decision-making tasks: in
simulated data with binary outcomes and in a medical dataset with synthetic and
continuous treatment outcomes.
