  Performance metrics for medical image segmentation models are used to measure
the agreement between the reference annotation and the predicted segmentation.
Usually, overlap metrics, such as the Dice, are used as a metric to evaluate
the performance of these models in order for results to be comparable. However,
there is a mismatch between the distributions of cases and difficulty level of
segmentation tasks in public data sets compared to clinical practice. Common
metrics fail to measure the impact of this mismatch, especially for clinical
data sets that include low signal pathologies, a difficult segmentation task,
and uncertain, small, or empty reference annotations. This limitation may
result in ineffective research of machine learning practitioners in designing
and optimizing models. Dimensions of evaluating clinical value include
consideration of the uncertainty of reference annotations, independence from
reference annotation volume size, and evaluation of classification of empty
reference annotations. We study how uncertain, small, and empty reference
annotations influence the value of metrics for medical image segmentation on an
in-house data set regardless of the model. We examine metrics behavior on the
predictions of a standard deep learning framework in order to identify metrics
with clinical value. We compare to a public benchmark data set (BraTS 2019)
with a high-signal pathology and certain, larger, and no empty reference
annotations. We may show machine learning practitioners, how uncertain, small,
or empty reference annotations require a rethinking of the evaluation and
optimizing procedures. The evaluation code was released to encourage further
analysis of this topic.
https://github.com/SophieOstmeier/UncertainSmallEmpty.git
