  Deep learning fostered a leap ahead in automated skin lesion analysis in the
last two years. Those models are expensive to train and difficult to
parameterize. Objective: We investigate methodological issues for designing and
evaluating deep learning models for skin lesion analysis. We explore 10 choices
faced by researchers: use of transfer learning, model architecture, train
dataset, image resolution, type of data augmentation, input normalization, use
of segmentation, duration of training, additional use of SVMs, and test data
augmentation. Methods: We perform two full factorial experiments, for five
different test datasets, resulting in 2560 exhaustive trials in our main
experiment, and 1280 trials in our assessment of transfer learning. We analyze
both with multi-way ANOVA. We use the exhaustive trials to simulate sequential
decisions and ensembles, with and without the use of privileged information
from the test set. Results -- main experiment: Amount of train data has
disproportionate influence, explaining almost half the variation in
performance. Of the other factors, test data augmentation and input resolution
are the most influential. Deeper models, when combined, with extra data, also
help. -- transfer experiment: Transfer learning is critical, its absence brings
huge performance penalties. -- simulations: Ensembles of models are the best
option to provide reliable results with limited resources, without using
privileged information and sacrificing methodological rigor. Conclusions and
Significance: Advancing research on automated skin lesion analysis requires
curating larger public datasets. Indirect use of privileged information from
the test set to design the models is a subtle, but frequent methodological
mistake that leads to overoptimistic results. Ensembles of models are a
cost-effective alternative to the expensive full-factorial and to the unstable
sequential designs.
