  Contrastive Language-Image Pre-training (CLIP) is a powerful multimodal large
vision model that has demonstrated significant benefits for downstream tasks,
including many zero-shot learning and text-guided vision tasks. However, we
notice some severe problems regarding the model's explainability, which
undermines its credibility and impedes related tasks. Specifically, we find
CLIP prefers the background regions than the foregrounds according to the
predicted similarity map, which contradicts human understanding. Besides, there
are obvious noisy activations on the visualization results at irrelevant
positions. To address these two issues, we conduct in-depth analyses and reveal
the reasons with new findings and evidences. Based on these insights, we
propose the CLIP Surgery, a method that enables surgery-like modifications for
the inference architecture and features, for better explainability and
enhancement in multiple open-vocabulary tasks. The proposed method has
significantly improved the explainability of CLIP for both convolutional
networks and vision transformers, surpassing existing methods by large margins.
Besides, our approach also demonstrates remarkable improvements in
open-vocabulary segmentation and multi-label recognition tasks. For examples,
the mAP improvement on NUS-Wide multi-label recognition is 4.41% without any
additional training, and our CLIP Surgery surpasses the state-of-the-art method
by 8.74% at mIoU on Cityscapes open-vocabulary semantic segmentation.
Furthermore, our method benefits other tasks including multimodal visualization
and interactive segmentation like Segment Anything Model (SAM). The code is
available at https://github.com/xmed-lab/CLIP_Surgery
