  The last decade of computer vision has been dominated by Deep Learning
architectures, thanks to their unparalleled success. Their performance,
however, often comes at the cost of explainability owing to their highly
non-linear nature. Consequently, a parallel field of eXplainable Artificial
Intelligence (XAI) has developed with the aim of generating insights regarding
the decision making process of deep learning models. An important problem in
XAI is that of the generation of saliency maps. These are regions in an input
image which contributed most towards the model's final decision. Most work in
this regard, however, has been focused on image classification, and image
segmentation - despite being a ubiquitous task - has not received the same
attention. In the present work, we propose MiSuRe (Minimally Sufficient Region)
as an algorithm to generate saliency maps for image segmentation. The goal of
the saliency maps generated by MiSuRe is to get rid of irrelevant regions, and
only highlight those regions in the input image which are crucial to the image
segmentation decision. We perform our analysis on 3 datasets: Triangle
(artificially constructed), COCO-2017 (natural images), and the Synapse
multi-organ (medical images). Additionally, we identify a potential usecase of
these post-hoc saliency maps in order to perform post-hoc reliability of the
segmentation model.
