  Many popular algorithmic fairness measures depend on the joint distribution
of predictions, outcomes, and a sensitive feature like race or gender. These
measures are sensitive to distribution shift: a predictor which is trained to
satisfy one of these fairness definitions may become unfair if the distribution
changes. In performative prediction settings, however, predictors are precisely
intended to induce distribution shift. For example, in many applications in
criminal justice, healthcare, and consumer finance, the purpose of building a
predictor is to reduce the rate of adverse outcomes such as recidivism,
hospitalization, or default on a loan. We formalize the effect of such
predictors as a type of concept shift-a particular variety of distribution
shift-and show both theoretically and via simulated examples how this causes
predictors which are fair when they are trained to become unfair when they are
deployed. We further show how many of these issues can be avoided by using
fairness definitions that depend on counterfactual rather than observable
outcomes.
