  Understanding different human attributes and how they affect model behavior
may become a standard need for all model creation and usage, from traditional
computer vision tasks to the newest multimodal generative AI systems. In
computer vision specifically, we have relied on datasets augmented with
perceived attribute signals (e.g., gender presentation, skin tone, and age) and
benchmarks enabled by these datasets. Typically labels for these tasks come
from human annotators. However, annotating attribute signals, especially skin
tone, is a difficult and subjective task. Perceived skin tone is affected by
technical factors, like lighting conditions, and social factors that shape an
annotator's lived experience. This paper examines the subjectivity of skin tone
annotation through a series of annotation experiments using the Monk Skin Tone
(MST) scale, a small pool of professional photographers, and a much larger pool
of trained crowdsourced annotators. Along with this study we release the Monk
Skin Tone Examples (MST-E) dataset, containing 1515 images and 31 videos spread
across the full MST scale. MST-E is designed to help train human annotators to
annotate MST effectively. Our study shows that annotators can reliably annotate
skin tone in a way that aligns with an expert in the MST scale, even under
challenging environmental conditions. We also find evidence that annotators
from different geographic regions rely on different mental models of MST
categories resulting in annotations that systematically vary across regions.
Given this, we advise practitioners to use a diverse set of annotators and a
higher replication count for each image when annotating skin tone for fairness
research.
