  Fairness is crucial when training a deep-learning discriminative model,
especially in the facial domain. Models tend to correlate specific
characteristics (such as age and skin color) with unrelated attributes
(downstream tasks), resulting in biases which do not correspond to reality. It
is common knowledge that these correlations are present in the data and are
then transferred to the models during training. This paper proposes a method to
mitigate these correlations to improve fairness. To do so, we learn
interpretable and meaningful paths lying in the semantic space of a pre-trained
diffusion model (DiffAE) -- such paths being supervised by contrastive text
dipoles. That is, we learn to edit protected characteristics (age and skin
color). These paths are then applied to augment images to improve the fairness
of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on
several downstream tasks with age and skin color as protected characteristics.
As a proxy for fairness, we compute the difference in accuracy with respect to
the protected characteristics. Quantitative results show how the augmented
images help the model improve the overall accuracy, the aforementioned metric,
and the disparity of equal opportunity. Code is available at:
https://github.com/Moreno98/Vision-Language-Bias-Control.
