  Concept Bottleneck Models (CBMs) provide interpretable prediction by
introducing an intermediate Concept Bottleneck Layer (CBL), which encodes
human-understandable concepts to explain models' decision. Recent works
proposed to utilize Large Language Models (LLMs) and pre-trained
Vision-Language Models (VLMs) to automate the training of CBMs, making it more
scalable and automated. However, existing approaches still fall short in two
aspects: First, the concepts predicted by CBL often mismatch the input image,
raising doubts about the faithfulness of interpretation. Second, it has been
shown that concept values encode unintended information: even a set of random
concepts could achieve comparable test accuracy to state-of-the-art CBMs. To
address these critical limitations, in this work, we propose a novel framework
called Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable
faithful interpretability with the benefits of boosted performance. Our method
leverages off-the-shelf open-domain grounded object detectors to provide
visually grounded concept annotation, which largely enhances the faithfulness
of concept prediction while further improving the model performance. In
addition, we propose a new metric called Number of Effective Concepts (NEC) to
control the information leakage and provide better interpretability. Extensive
evaluations across five standard benchmarks show that our method, VLG-CBM,
outperforms existing methods by at least 4.27% and up to 51.09% on accuracy at
NEC=5, and by at least 0.45% and up to 29.78% on average accuracy across
different NECs, while preserves both faithfulness and interpretability of the
learned concepts as demonstrated in extensive experiments.
