  Faces form the basis for a rich variety of judgments in humans, yet the
underlying features remain poorly understood. Although fine-grained
distinctions within a race might more strongly constrain possible facial
features used by humans than in case of coarse categories such as race or
gender, such fine grained distinctions are relatively less studied.
Fine-grained race classification is also interesting because even humans may
not be perfectly accurate on these tasks. This allows us to compare errors made
by humans and machines, in contrast to standard object detection tasks where
human performance is nearly perfect. We have developed a novel face database of
close to 1650 diverse Indian faces labeled for fine-grained race (South vs
North India) as well as for age, weight, height and gender. We then asked close
to 130 human subjects who were instructed to categorize each face as belonging
toa Northern or Southern state in India. We then compared human performance on
this task with that of computational models trained on the ground-truth labels.
Our main results are as follows: (1) Humans are highly consistent (average
accuracy: 63.6%), with some faces being consistently classified with > 90%
accuracy and others consistently misclassified with < 30% accuracy; (2) Models
trained on ground-truth labels showed slightly worse performance (average
accuracy: 62%) but showed higher accuracy (72.2%) on faces classified with >
80% accuracy by humans. This was true for models trained on simple spatial and
intensity measurements extracted from faces as well as deep neural networks
trained on race or gender classification; (3) Using overcomplete banks of
features derived from each face part, we found that mouth shape was the single
largest contributor towards fine-grained race classification, whereas distances
between face parts was the strongest predictor of gender.
