  Predictive algorithms are now used to help distribute a large share of our
society's resources and sanctions, such as healthcare, loans, criminal
detentions, and tax audits. Under the right circumstances, these algorithms can
improve the efficiency and equity of decision-making. At the same time, there
is a danger that the algorithms themselves could entrench and exacerbate
disparities, particularly along racial, ethnic, and gender lines. To help
ensure their fairness, many researchers suggest that algorithms be subject to
at least one of three constraints: (1) no use of legally protected features,
such as race, ethnicity, and gender; (2) equal rates of "positive" decisions
across groups; and (3) equal error rates across groups. Here we show that these
constraints, while intuitively appealing, often worsen outcomes for individuals
in marginalized groups, and can even leave all groups worse off. The inherent
trade-off we identify between formal fairness constraints and welfare
improvements -- particularly for the marginalized -- highlights the need for a
more robust discussion on what it means for an algorithm to be "fair". We
illustrate these ideas with examples from healthcare and the criminal-legal
system, and make several proposals to help practitioners design more equitable
algorithms.
