  Machine Learning (ML) algorithms shape our lives. Banks use them to determine
if we are good borrowers; IT companies delegate them recruitment decisions;
police apply ML for crime-prediction, and judges base their verdicts on ML.
However, real-world examples show that such automated decisions tend to
discriminate against protected groups. This potential discrimination generated
a huge hype both in media and in the research community. Quite a few formal
notions of fairness were proposed, which take a form of constraints a "fair"
algorithm must satisfy. We focus on scenarios where fairness is imposed on a
self-interested party (e.g., a bank that maximizes its revenue). We find that
the disadvantaged protected group can be worse off after imposing a fairness
constraint. We introduce a family of \textit{Welfare-Equalizing} fairness
constraints that equalize per-capita welfare of protected groups, and include
\textit{Demographic Parity} and \textit{Equal Opportunity} as particular cases.
In this family, we characterize conditions under which the fairness constraint
helps the disadvantaged group. We also characterize the structure of the
optimal \textit{Welfare-Equalizing} classifier for the self-interested party,
and provide an algorithm to compute it. Overall, our
\textit{Welfare-Equalizing} fairness approach provides a unified framework for
discussing fairness in classification in the presence of a self-interested
party.
