  Face recognition performance has seen a tremendous gain in recent years,
mostly due to the availability of large-scale face images dataset that can be
exploited by deep neural networks to learn powerful face representations.
However, recent research has shown differences in face recognition performance
across different ethnic groups mostly due to the racial imbalance in the
training datasets where Caucasian identities largely dominate other
ethnicities. This is actually symptomatic of the under-representation of
non-Caucasian ethnic groups in the celebdom from which face datasets are
usually gathered, rendering the acquisition of labeled data of the
under-represented groups challenging. In this paper, we propose an Asymmetric
Rejection Loss, which aims at making full use of unlabeled images of those
under-represented groups, to reduce the racial bias of face recognition models.
We view each unlabeled image as a unique class, however as we cannot guarantee
that two unlabeled samples are from a distinct class we exploit both labeled
and unlabeled data in an asymmetric manner in our loss formalism. Extensive
experiments show our method's strength in mitigating racial bias, outperforming
state-of-the-art semi-supervision methods. Performance on the under-represented
ethnicity groups increases while that on the well-represented group is nearly
unchanged.
