  Neural networks achieve the state-of-the-art in image classification tasks.
However, they can encode spurious variations or biases that may be present in
the training data. For example, training an age predictor on a dataset that is
not balanced for gender can lead to gender biased predicitons (e.g. wrongly
predicting that males are older if only elderly males are in the training set).
We present two distinct contributions: 1) An algorithm that can remove multiple
sources of variation from the feature representation of a network. We
demonstrate that this algorithm can be used to remove biases from the feature
representation, and thereby improve classification accuracies, when training
networks on extremely biased datasets. 2) An ancestral origin database of
14,000 images of individuals from East Asia, the Indian subcontinent,
sub-Saharan Africa, and Western Europe. We demonstrate on this dataset, for a
number of facial attribute classification tasks, that we are able to remove
racial biases from the network feature representation.
