  We explore the problem of learning under selective labels in the context of
algorithm-assisted decision making. Selective labels is a pervasive selection
bias problem that arises when historical decision making blinds us to the true
outcome for certain instances. Examples of this are common in many
applications, ranging from predicting recidivism using pre-trial release data
to diagnosing patients. In this paper we discuss why selective labels often
cannot be effectively tackled by standard methods for adjusting for sample
selection bias, even if there are no unobservables. We propose a data
augmentation approach that can be used to either leverage expert consistency to
mitigate the partial blindness that results from selective labels, or to
empirically validate whether learning under such framework may lead to
unreliable models prone to systemic discrimination.
