  Explainability is an aspect of modern AI that is vital for impact and
usability in the real world. The main objective of this paper is to emphasise
the need to understand the predictions of Computer Vision models, specifically
Convolutional Neural Network (CNN) based models. Existing methods of explaining
CNN predictions are mostly based on Gradient-weighted Class Activation Maps
(Grad-CAM) and solely focus on a single target class. We show that from the
point of the target class selection, we make an assumption on the prediction
process, hence neglecting a large portion of the predictor CNN model's thinking
process. In this paper, we present an exhaustive methodology called Fused
Multi-class Gradient-weighted Class Activation Map (FM-G-CAM) that considers
multiple top predicted classes, which provides a holistic explanation of the
predictor CNN's thinking rationale. We also provide a detailed and
comprehensive mathematical and algorithmic description of our method.
Furthermore, along with a concise comparison of existing methods, we compare
FM-G-CAM with Grad-CAM, highlighting its benefits through real-world practical
use cases. Finally, we present an open-source Python library with FM-G-CAM
implementation to conveniently generate saliency maps for CNN-based model
predictions.
