  We report on the results of a small crowdsourcing experiment conducted at a
workshop on machine learning for segmentation held at the Danish Bio Imaging
network meeting 2020. During the workshop we asked participants to manually
segment mitochondria in three 2D patches. The aim of the experiment was to
illustrate that manual annotations should not be seen as the ground truth, but
as a reference standard that is subject to substantial variation. In this note
we show how the large variation we observed in the segmentations can be reduced
by removing the annotators with worst pair-wise agreement. Having removed the
annotators with worst performance, we illustrate that the remaining variance is
semantically meaningful and can be exploited to obtain segmentations of cell
boundary and cell interior.
