  Although deep face recognition has achieved impressive progress in recent
years, controversy has arisen regarding discrimination based on skin tone,
questioning their deployment into real-world scenarios. In this paper, we aim
to systematically and scientifically study this bias from both data and
algorithm aspects. First, using the dermatologist approved Fitzpatrick Skin
Type classification system and Individual Typology Angle, we contribute a
benchmark called Identity Shades (IDS) database, which effectively quantifies
the degree of the bias with respect to skin tone in existing face recognition
algorithms and commercial APIs. Further, we provide two skin-tone aware
training datasets, called BUPT-Globalface dataset and BUPT-Balancedface
dataset, to remove bias in training data. Finally, to mitigate the algorithmic
bias, we propose a novel meta-learning algorithm, called Meta Balanced Network
(MBN), which learns adaptive margins in large margin loss such that the model
optimized by this loss can perform fairly across people with different skin
tones. To determine the margins, our method optimizes a meta skewness loss on a
clean and unbiased meta set and utilizes backward-on-backward automatic
differentiation to perform a second order gradient descent step on the current
margins. Extensive experiments show that MBN successfully mitigates bias and
learns more balanced performance for people with different skin tones in face
recognition. The proposed datasets are available at
http://www.whdeng.cn/RFW/index.html.
