  Automated gender classification has important applications in many domains,
such as demographic research, law enforcement, online advertising, as well as
human-computer interaction. Recent research has questioned the fairness of this
technology across gender and race. Specifically, the majority of the studies
raised the concern of higher error rates of the face-based gender
classification system for darker-skinned people like African-American and for
women. However, to date, the majority of existing studies were limited to
African-American and Caucasian only. The aim of this paper is to investigate
the differential performance of the gender classification algorithms across
gender-race groups. To this aim, we investigate the impact of (a) architectural
differences in the deep learning algorithms and (b) training set imbalance, as
a potential source of bias causing differential performance across gender and
race. Experimental investigations are conducted on two latest large-scale
publicly available facial attribute datasets, namely, UTKFace and FairFace. The
experimental results suggested that the algorithms with architectural
differences varied in performance with consistency towards specific gender-race
groups. For instance, for all the algorithms used, Black females (Black race in
general) always obtained the least accuracy rates. Middle Eastern males and
Latino females obtained higher accuracy rates most of the time. Training set
imbalance further widens the gap in the unequal accuracy rates across all
gender-race groups. Further investigations using facial landmarks suggested
that facial morphological differences due to the bone structure influenced by
genetic and environmental factors could be the cause of the least performance
of Black females and Black race, in general.
