  CLIP yielded impressive results on zero-shot transfer learning tasks and is
considered as a foundation model like BERT or GPT3. CLIP vision models that
have a rich representation are pre-trained using the InfoNCE objective and
natural language supervision before they are fine-tuned on particular tasks.
Though CLIP excels at zero-shot transfer learning, it suffers from an
explaining away problem, that is, it focuses on one or few features, while
neglecting other relevant features. This problem is caused by insufficiently
extracting the covariance structure in the original multi-modal data. We
suggest to use modern Hopfield networks to tackle the problem of explaining
away. Their retrieved embeddings have an enriched covariance structure derived
from co-occurrences of features in the stored embeddings. However, modern
Hopfield networks increase the saturation effect of the InfoNCE objective which
hampers learning. We propose to use the InfoLOOB objective to mitigate this
saturation effect. We introduce the novel "Contrastive Leave One Out Boost"
(CLOOB), which uses modern Hopfield networks for covariance enrichment together
with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after
pre-training on the Conceptual Captions and the YFCC dataset with respect to
their zero-shot transfer learning performance on other datasets. CLOOB
consistently outperforms CLIP at zero-shot transfer learning across all
considered architectures and datasets.
