  The issue of demographic disparities in face recognition accuracy has
attracted increasing attention in recent years. Various face image datasets
have been proposed as 'fair' or 'balanced' to assess the accuracy of face
recognition algorithms across demographics. These datasets typically balance
the number of identities and images across demographics. It is important to
note that the number of identities and images in an evaluation dataset are {\em
not} driving factors for 1-to-1 face matching accuracy. Moreover, balancing the
number of identities and images does not ensure balance in other factors known
to impact accuracy, such as head pose, brightness, and image quality. We
demonstrate these issues using several recently proposed datasets. To improve
the ability to perform less biased evaluations, we propose a bias-aware toolkit
that facilitates creation of cross-demographic evaluation datasets balanced on
factors mentioned in this paper.
