  Dynamic learning systems subject to selective labeling exhibit censoring,
i.e. persistent negative predictions assigned to one or more subgroups of
points. In applications like consumer finance, this results in groups of
applicants that are persistently denied and thus never enter into the training
data. In this work, we formalize censoring, demonstrate how it can arise, and
highlight difficulties in detection. We consider safeguards against censoring -
recourse and randomized-exploration - both of which ensure we collect labels
for points that would otherwise go unobserved. The resulting techniques allow
examples from censored groups to enter into the training data and correct the
model. Our results highlight the otherwise unmeasured harms of censoring and
demonstrate the effectiveness of mitigation strategies across a range of data
generating processes.
