  Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP,
have achieved a remarkable success in learning representations that are robust
against distribution shift and generalize to new domains. Despite the empirical
success, the mechanism behind learning such generalizable representations is
not understood. In this work, we rigorously analyze this problem and uncover
two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which
allows the model to learn features with a high variance, and \emph{inter-class
feature sharing}, where annotated details in one class help learning other
classes better. Both mechanisms prevent spurious features that are
over-represented in the training data to overshadow the generalizable core
features. This yields superior zero-shot classification accuracy under
distribution shift. Furthermore, we theoretically demonstrate the benefits of
using rich captions on robustness and explore the effect of annotating
different types of details in the captions. We validate our theoretical
findings through experiments, including a well-designed synthetic experiment
and an experiment involving training CLIP models on MSCOCO/Conceptual Captions
and evaluating them on shifted ImageNets.
