  Recent advances in computer vision and deep learning have led to
breakthroughs in the development of automated skin image analysis. In
particular, skin cancer classification models have achieved performance higher
than trained expert dermatologists. However, no attempt has been made to
evaluate the consistency in performance of machine learning models across
populations with varying skin tones. In this paper, we present an approach to
estimate skin tone in benchmark skin disease datasets, and investigate whether
model performance is dependent on this measure. Specifically, we use individual
typology angle (ITA) to approximate skin tone in dermatology datasets. We look
at the distribution of ITA values to better understand skin color
representation in two benchmark datasets: 1) the ISIC 2018 Challenge dataset, a
collection of dermoscopic images of skin lesions for the detection of skin
cancer, and 2) the SD-198 dataset, a collection of clinical images capturing a
wide variety of skin diseases. To estimate ITA, we first develop segmentation
models to isolate non-diseased areas of skin. We find that the majority of the
data in the the two datasets have ITA values between 34.5{\deg} and 48{\deg},
which are associated with lighter skin, and is consistent with
under-representation of darker skinned populations in these datasets. We also
find no measurable correlation between performance of machine learning model
and ITA values, though more comprehensive data is needed for further
validation.
