  Many existing fairness criteria for machine learning involve equalizing some
metric across protected groups such as race or gender. However, practitioners
trying to audit or enforce such group-based criteria can easily face the
problem of noisy or biased protected group information. First, we study the
consequences of naively relying on noisy protected group labels: we provide an
upper bound on the fairness violations on the true groups G when the fairness
criteria are satisfied on noisy groups $\hat{G}$. Second, we introduce two new
approaches using robust optimization that, unlike the naive approach of only
relying on $\hat{G}$, are guaranteed to satisfy fairness criteria on the true
protected groups G while minimizing a training objective. We provide
theoretical guarantees that one such approach converges to an optimal feasible
solution. Using two case studies, we show empirically that the robust
approaches achieve better true group fairness guarantees than the naive
approach.
