  Recent work in fairness in machine learning has proposed adjusting for
fairness by equalizing accuracy metrics across groups and has also studied how
datasets affected by historical prejudices may lead to unfair decision
policies. We connect these lines of work and study the residual unfairness that
arises when a fairness-adjusted predictor is not actually fair on the target
population due to systematic censoring of training data by existing biased
policies. This scenario is particularly common in the same applications where
fairness is a concern. We characterize theoretically the impact of such
censoring on standard fairness metrics for binary classifiers and provide
criteria for when residual unfairness may or may not appear. We prove that,
under certain conditions, fairness-adjusted classifiers will in fact induce
residual unfairness that perpetuates the same injustices, against the same
groups, that biased the data to begin with, thus showing that even
state-of-the-art fair machine learning can have a "bias in, bias out" property.
When certain benchmark data is available, we show how sample reweighting can
estimate and adjust fairness metrics while accounting for censoring. We use
this to study the case of Stop, Question, and Frisk (SQF) and demonstrate that
attempting to adjust for fairness perpetuates the same injustices that the
policy is infamous for.
