  With increasing adoption of face recognition systems, it is important to
ensure adequate performance of these technologies across demographic groups.
Recently, phenotypes such as skin-tone, have been proposed as superior
alternatives to traditional race categories when exploring performance
differentials. However, there is little consensus regarding how to
appropriately measure skin-tone in evaluations of biometric performance or in
AI more broadly. In this study, we explore the relationship between
face-area-lightness-measures (FALMs) estimated from images and ground-truth
skin readings collected using a device designed to measure human skin. FALMs
estimated from different images of the same individual varied significantly
relative to ground-truth FALM. This variation was only reduced by greater
control of acquisition (camera, background, and environment). Next, we compare
ground-truth FALM to Fitzpatrick Skin Types (FST) categories obtained using the
standard, in-person, medical survey and show FST is poorly predictive of
skin-tone. Finally, we show how noisy estimation of FALM leads to errors
selecting explanatory factors for demographic differentials. These results
demonstrate that measures of skin-tone for biometric performance evaluations
must come from objective, characterized, and controlled sources. Further,
despite this being a currently practiced approach, estimating FST categories
and FALMs from uncontrolled imagery does not provide an appropriate measure of
skin-tone.
