  Machine learning systems produce biased results towards certain demographic
groups, known as the fairness problem. Recent approaches to tackle this problem
learn a latent code (i.e., representation) through disentangled representation
learning and then discard the latent code dimensions correlated with sensitive
attributes (e.g., gender). Nevertheless, these approaches may suffer from
incomplete disentanglement and overlook proxy attributes (proxies for sensitive
attributes) when processing real-world data, especially for unstructured data,
causing performance degradation in fairness and loss of useful information for
downstream tasks. In this paper, we propose a novel fairness framework that
performs debiasing with regard to both sensitive attributes and proxy
attributes, which boosts the prediction performance of downstream task models
without complete disentanglement. The main idea is to, first, leverage
gradient-based explanation to find two model focuses, 1) one focus for
predicting sensitive attributes and 2) the other focus for predicting
downstream task labels, and second, use them to perturb the latent code that
guides the training of downstream task models towards fairness and utility
goals. We show empirically that our framework works with both disentangled and
non-disentangled representation learning methods and achieves better
fairness-accuracy trade-off on unstructured and structured datasets than
previous state-of-the-art approaches.
