  To interpret deep learning models, one mainstream is to explore the learned
concepts by networks. Testing with Concept Activation Vector (TCAV) presents a
powerful tool to quantify the contribution of query concepts (represented by
user-defined guidance images) to a target class. For example, we can
quantitatively evaluate whether and to what extent concept striped contributes
to model prediction zebra with TCAV. Therefore, TCAV whitens the reasoning
process of deep networks. And it has been applied to solve practical problems
such as diagnosis. However, for some images where the target object only
occupies a small fraction of the region, TCAV evaluation may be interfered with
by redundant background features because TCAV calculates concept contribution
to a target class based on a whole hidden layer. To tackle this problem, based
on TCAV, we propose Spatial Activation Concept Vector (SACV) which identifies
the relevant spatial locations to the query concept while evaluating their
contributions to the model prediction of the target class. Experiment shows
that SACV generates a more fine-grained explanation map for a hidden layer and
quantifies concepts' contributions spatially. Moreover, it avoids interference
from background features. The code is available on
https://github.com/AntonotnaWang/Spatial-Activation-Concept-Vector.
