  Human-centric explainability of AI-based Decision Support Systems (DSS) using
visual input modalities is directly related to reliability and practicality of
such algorithms. An otherwise accurate and robust DSS might not enjoy trust of
experts in critical application areas if it is not able to provide reasonable
justification of its predictions. This paper introduces Concept Localization
Maps (CLMs), which is a novel approach towards explainable image classifiers
employed as DSS. CLMs extend Concept Activation Vectors (CAVs) by locating
significant regions corresponding to a learned concept in the latent space of a
trained image classifier. They provide qualitative and quantitative assurance
of a classifier's ability to learn and focus on similar concepts important for
humans during image recognition. To better understand the effectiveness of the
proposed method, we generated a new synthetic dataset called Simple Concept
DataBase (SCDB) that includes annotations for 10 distinguishable concepts, and
made it publicly available. We evaluated our proposed method on SCDB as well as
a real-world dataset called CelebA. We achieved localization recall of above
80% for most relevant concepts and average recall above 60% for all concepts
using SE-ResNeXt-50 on SCDB. Our results on both datasets show great promise of
CLMs for easing acceptance of DSS in practice.
