  Pre-trained large-scale models have become a major research focus, but their
effectiveness is limited in real-world applications due to diverse data
distributions. In contrast, humans excel at decision-making across various
domains by learning reusable knowledge that remains invariant despite
environmental changes in a complex world. Although CLIP, as a successful
vision-language pre-trained model, demonstrates remarkable performance in
various visual downstream tasks, our experiments reveal unsatisfactory results
in specific domains. Our further analysis with causal inference exposes the
current CLIP model's inability to capture the invariant causal mechanisms
across domains, attributed to its deficiency in identifying latent factors
generating the data. To address this, we propose the Invariant Causal Mechanism
of CLIP (CLIP-ICM), an algorithm designed to provably identify invariant latent
factors with the aid of interventional data, and perform accurate prediction on
various domains. Theoretical analysis demonstrates that our method has a lower
generalization bound in out-of-distribution (OOD) scenarios. Experimental
results showcase the outstanding performance of CLIP-ICM.
