  To equip Convolutional Neural Networks (CNNs) with explainability, it is
essential to interpret how opaque models take specific decisions, understand
what causes the errors, improve the architecture design, and identify unethical
biases in the classifiers. This paper introduces ADVISE, a new explainability
method that quantifies and leverages the relevance of each unit of the feature
map to provide better visual explanations. To this end, we propose using
adaptive bandwidth kernel density estimation to assign a relevance score to
each unit of the feature map with respect to the predicted class. We also
propose an evaluation protocol to quantitatively assess the visual
explainability of CNN models. We extensively evaluate our idea in the image
classification task using AlexNet, VGG16, ResNet50, and Xception pretrained on
ImageNet. We compare ADVISE with the state-of-the-art visual explainable
methods and show that the proposed method outperforms competing approaches in
quantifying feature-relevance and visual explainability while maintaining
competitive time complexity. Our experiments further show that ADVISE fulfils
the sensitivity and implementation independence axioms while passing the sanity
checks. The implementation is accessible for reproducibility purposes on
https://github.com/dehshibi/ADVISE.
