  Learning visual representation of high quality is essential for image
classification. Recently, a series of contrastive representation learning
methods have achieved preeminent success. Particularly, SupCon outperformed the
dominant methods based on cross-entropy loss in representation learning.
However, we notice that there could be potential ethical risks in supervised
contrastive learning. In this paper, we for the first time analyze unfairness
caused by supervised contrastive learning and propose a new Fair Supervised
Contrastive Loss (FSCL) for fair visual representation learning. Inheriting the
philosophy of supervised contrastive learning, it encourages representation of
the same class to be closer to each other than that of different classes, while
ensuring fairness by penalizing the inclusion of sensitive attribute
information in representation. In addition, we introduce a group-wise
normalization to diminish the disparities of intra-group compactness and
inter-class separability between demographic groups that arouse unfair
classification. Through extensive experiments on CelebA and UTK Face, we
validate that the proposed method significantly outperforms SupCon and existing
state-of-the-art methods in terms of the trade-off between top-1 accuracy and
fairness. Moreover, our method is robust to the intensity of data bias and
effectively works in incomplete supervised settings. Our code is available at
https://github.com/sungho-CoolG/FSCL.
