  Learning good representations involves capturing the diverse ways in which
data samples relate. Contrastive loss - an objective matching related samples -
underlies methods from self-supervised to multimodal learning. Contrastive
losses, however, can be viewed more broadly as modifying a similarity graph to
indicate how samples should relate in the embedding space. This view reveals a
shortcoming in contrastive learning: the similarity graph is binary, as only
one sample is the related positive sample. Crucially, similarities
\textit{across} samples are ignored. Based on this observation, we revise the
standard contrastive loss to explicitly encode how a sample relates to others.
We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive,
to train vision models based on similarities in class or text caption
descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M
with 3 million, and CC12M with 12 million samples. The representations learned
via our objective outperform both contrastive self-supervised and
vision-language models trained on the same data across a range of tasks. When
training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet
Real. Our objective appears to work particularly well in lower-data regimes,
with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when
training with CC3M. Finally, our objective seems to encourage the model to
learn representations that separate objects from their attributes and
backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the
proposed solution takes a small step towards developing richer learning
objectives for understanding sample relations in foundation models.
