  Convolutional neural networks have become state-of-the-art in a wide range of
image recognition tasks. The interpretation of their predictions, however, is
an active area of research. Whereas various interpretation methods have been
suggested for image classification, the interpretation of image segmentation
still remains largely unexplored. To that end, we propose SEG-GRAD-CAM, a
gradient-based method for interpreting semantic segmentation. Our method is an
extension of the widely-used Grad-CAM method, applied locally to produce
heatmaps showing the relevance of individual pixels for semantic segmentation.
