  Motivated by settings in which predictive models may be required to be
non-discriminatory with respect to certain attributes (such as race), but even
collecting the sensitive attribute may be forbidden or restricted, we initiate
the study of fair learning under the constraint of differential privacy. We
design two learning algorithms that simultaneously promise differential privacy
and equalized odds, a 'fairness' condition that corresponds to equalizing false
positive and negative rates across protected groups. Our first algorithm is a
private implementation of the equalized odds post-processing approach of [Hardt
et al., 2016]. This algorithm is appealingly simple, but must be able to use
protected group membership explicitly at test time, which can be viewed as a
form of 'disparate treatment'. Our second algorithm is a differentially private
version of the oracle-efficient in-processing approach of [Agarwal et al.,
2018] that can be used to find the optimal fair classifier, given access to a
subroutine that can solve the original (not necessarily fair) learning problem.
This algorithm is more complex but need not have access to protected group
membership at test time. We identify new tradeoffs between fairness, accuracy,
and privacy that emerge only when requiring all three properties, and show that
these tradeoffs can be milder if group membership may be used at test time. We
conclude with a brief experimental evaluation.
