  In this work, we explore the issue of the inter-annotator agreement for
training and evaluating automated segmentation of skin lesions. We explore what
different degrees of agreement represent, and how they affect different use
cases for segmentation. We also evaluate how conditioning the ground truths
using different (but very simple) algorithms may help to enhance agreement and
may be appropriate for some use cases. The segmentation of skin lesions is a
cornerstone task for automated skin lesion analysis, useful both as an
end-result to locate/detect the lesions and as an ancillary task for lesion
classification. Lesion segmentation, however, is a very challenging task, due
not only to the challenge of image segmentation itself but also to the
difficulty in obtaining properly annotated data. Detecting accurately the
borders of lesions is challenging even for trained humans, since, for many
lesions, those borders are fuzzy and ill-defined. Using lesions and annotations
from the ISIC Archive, we estimate inter-annotator agreement for skin-lesion
segmentation and propose several simple procedures that may help to improve
inter-annotator agreement if used to condition the ground truths.
