  Semantic segmentation has become an important task in computer vision with
the growth of self-driving cars, medical image segmentation, etc. Although
current models provide excellent results, they are still far from perfect and
while there has been significant work in trying to improve the performance,
both with respect to accuracy and speed of segmentation, there has been little
work which analyses the failure cases of such systems. In this work, we aim to
provide an analysis of how segmentation fails across different models and
consider the question of whether these can be predicted reasonably at test
time. To do so, we explore existing uncertainty-based metrics and see how well
they correlate with misclassifications, allowing us to define the degree of
trust we put in the output of our prediction models. Through several
experiments on three different models across three datasets, we show that
simple measures such as entropy can be used to capture misclassification with
high recall rates.
