  The integration of vision-language models such as CLIP and Concept Bottleneck
Models (CBMs) offers a promising approach to explaining deep neural network
(DNN) decisions using concepts understandable by humans, addressing the
black-box concern of DNNs. While CLIP provides both explainability and
zero-shot classification capability, its pre-training on generic image and text
data may limit its classification accuracy and applicability to medical image
diagnostic tasks, creating a transfer learning problem. To maintain
explainability and address transfer learning needs, CBM methods commonly design
post-processing modules after the bottleneck module. However, this way has been
ineffective. This paper takes an unconventional approach by re-examining the
CBM framework through the lens of its geometrical representation as a simple
linear classification system. The analysis uncovers that post-CBM fine-tuning
modules merely rescale and shift the classification outcome of the system,
failing to fully leverage the system's learning potential. We introduce an
adaptive module strategically positioned between CLIP and CBM to bridge the gap
between source and downstream domains. This simple yet effective approach
enhances classification performance while preserving the explainability
afforded by the framework. Our work offers a comprehensive solution that
encompasses the entire process, from concept discovery to model training,
providing a holistic recipe for leveraging the strengths of GPT, CLIP, and CBM.
