  Fair face recognition is all about learning invariant feature that
generalizes to unseen faces in any demographic group. Unfortunately, face
datasets inevitably capture the imbalanced demographic attributes that are
ubiquitous in real-world observations, and the model learns biased feature that
generalizes poorly in the minority group. We point out that the bias arises due
to the confounding demographic attributes, which mislead the model to capture
the spurious demographic-specific feature. The confounding effect can only be
removed by causal intervention, which requires the confounder annotations.
However, such annotations can be prohibitively expensive due to the diversity
of the demographic attributes. To tackle this, we propose to generate diverse
data partitions iteratively in an unsupervised fashion. Each data partition
acts as a self-annotated confounder, enabling our Invariant Feature
Regularization (INV-REG) to deconfound. INV-REG is orthogonal to existing
methods, and combining INV-REG with two strong baselines (Arcface and CIFP)
leads to new state-of-the-art that improves face recognition on a variety of
demographic groups. Code is available at
https://github.com/PanasonicConnect/InvReg.
