  Metrics optimized in complex machine learning tasks are often selected in an
ad-hoc manner. It is unknown how they align with human expert perception. We
explore the correlations between established quantitative segmentation quality
metrics and qualitative evaluations by professionally trained human raters.
Therefore, we conduct psychophysical experiments for two complex biomedical
semantic segmentation problems. We discover that current standard metrics and
loss functions correlate only moderately with the segmentation quality
assessment of experts. Importantly, this effect is particularly pronounced for
clinically relevant structures, such as the enhancing tumor compartment of
glioma in brain magnetic resonance and grey matter in ultrasound imaging. It is
often unclear how to optimize abstract metrics, such as human expert
perception, in convolutional neural network (CNN) training. To cope with this
challenge, we propose a novel strategy employing techniques of classical
statistics to create complementary compound loss functions to better
approximate human expert perception. Across all rating experiments, human
experts consistently scored computer-generated segmentations better than the
human-curated reference labels. Our results, therefore, strongly question many
current practices in medical image segmentation and provide meaningful cues for
future research.
