  The limited ability of Convolutional Neural Networks to generalize to images
from previously unseen domains is a major limitation, in particular, for
safety-critical clinical tasks such as dermoscopic skin cancer classification.
In order to translate CNN-based applications into the clinic, it is essential
that they are able to adapt to domain shifts. Such new conditions can arise
through the use of different image acquisition systems or varying lighting
conditions. In dermoscopy, shifts can also occur as a change in patient age or
occurence of rare lesion localizations (e.g. palms). These are not prominently
represented in most training datasets and can therefore lead to a decrease in
performance. In order to verify the generalizability of classification models
in real world clinical settings it is crucial to have access to data which
mimics such domain shifts. To our knowledge no dermoscopic image dataset exists
where such domain shifts are properly described and quantified. We therefore
grouped publicly available images from ISIC archive based on their metadata
(e.g. acquisition location, lesion localization, patient age) to generate
meaningful domains. To verify that these domains are in fact distinct, we used
multiple quantification measures to estimate the presence and intensity of
domain shifts. Additionally, we analyzed the performance on these domains with
and without an unsupervised domain adaptation technique. We observed that in
most of our grouped domains, domain shifts in fact exist. Based on our results,
we believe these datasets to be helpful for testing the generalization
capabilities of dermoscopic skin cancer classifiers.
