  Vision-language pretrained models have seen remarkable success, but their
application to safety-critical settings is limited by their lack of
interpretability. To improve the interpretability of vision-language models
such as CLIP, we propose a multi-modal information bottleneck (M2IB) approach
that learns latent representations that compress irrelevant information while
preserving relevant visual and textual features. We demonstrate how M2IB can be
applied to attribution analysis of vision-language pretrained models,
increasing attribution accuracy and improving the interpretability of such
models when applied to safety-critical domains such as healthcare. Crucially,
unlike commonly used unimodal attribution methods, M2IB does not require ground
truth labels, making it possible to audit representations of vision-language
pretrained models when multiple modalities but no ground-truth data is
available. Using CLIP as an example, we demonstrate the effectiveness of M2IB
attribution and show that it outperforms gradient-based, perturbation-based,
and attention-based attribution methods both qualitatively and quantitatively.
