  We study the problem of performing classification in a manner that is fair
for sensitive groups, such as race and gender. This problem is tackled through
the lens of disentangled and locally fair representations. We learn a locally
fair representation, such that, under the learned representation, the
neighborhood of each sample is balanced in terms of the sensitive attribute.
For instance, when a decision is made to hire an individual, we ensure that the
$K$ most similar hired individuals are racially balanced. Crucially, we ensure
that similar individuals are found based on attributes not correlated to their
race. To this end, we disentangle the embedding space into two representations.
The first of which is correlated with the sensitive attribute while the second
is not. We apply our local fairness objective only to the second, uncorrelated,
representation. Through a set of experiments, we demonstrate the necessity of
both disentangled and local fairness for obtaining fair and accurate
representations. We evaluate our method on real-world settings such as
predicting income and re-incarceration rate and demonstrate the advantage of
our method.
