  Interpreting the decisions of deep learning models has been actively studied
since the explosion of deep neural networks. One of the most convincing
interpretation approaches is salience-based visual interpretation, such as
Grad-CAM, where the generation of attention maps depends merely on categorical
labels. Although existing interpretation methods can provide explainable
decision clues, they often yield partial correspondence between image and
saliency maps due to the limited discriminative information from one-hot
labels. This paper develops a Language-Image COnsistency model for explainable
image classification, termed LICO, by correlating learnable linguistic prompts
with corresponding visual features in a coarse-to-fine manner. Specifically, we
first establish a coarse global manifold structure alignment by minimizing the
distance between the distributions of image and language features. We then
achieve fine-grained saliency maps by applying optimal transport (OT) theory to
assign local feature maps with class-specific prompts. Extensive experimental
results on eight benchmark datasets demonstrate that the proposed LICO achieves
a significant improvement in generating more explainable attention maps in
conjunction with existing interpretation methods such as Grad-CAM. Remarkably,
LICO improves the classification performance of existing models without
introducing any computational overhead during inference. Source code is made
available at https://github.com/ymLeiFDU/LICO.
