  Learning fair representation is crucial for achieving fairness or debiasing
sensitive information. Most existing works rely on adversarial representation
learning to inject some invariance into representation. However, adversarial
learning methods are known to suffer from relatively unstable training, and
this might harm the balance between fairness and predictiveness of
representation. We propose a new approach, learning FAir Representation via
distributional CONtrastive Variational AutoEncoder (FarconVAE), which induces
the latent space to be disentangled into sensitive and nonsensitive parts. We
first construct the pair of observations with different sensitive attributes
but with the same labels. Then, FarconVAE enforces each non-sensitive latent to
be closer, while sensitive latents to be far from each other and also far from
the non-sensitive latent by contrasting their distributions. We provide a new
type of contrastive loss motivated by Gaussian and Student-t kernels for
distributional contrastive learning with theoretical analysis. Besides, we
adopt a new swap-reconstruction loss to boost the disentanglement further.
FarconVAE shows superior performance on fairness, pretrained model debiasing,
and domain generalization tasks from various modalities, including tabular,
image, and text.
