  We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach,
to reduce the memory consumption of contrastive loss when training contrastive
learning models. Our approach decomposes the contrastive loss and its gradient
computation into two parts, one to calculate the intra-GPU gradients and the
other to compute the inter-GPU gradients. According to our decomposition, only
the intra-GPU gradients are computed on the current GPU, while the inter-GPU
gradients are collected via all_reduce from other GPUs instead of being
repeatedly computed on every GPU. In this way, we can reduce the GPU memory
consumption of contrastive loss computation from $\bigO(B^2)$ to
$\bigO(\frac{B^2}{N})$, where $B$ and $N$ are the batch size and the number of
GPUs used for training. Such a distributed solution is mathematically
equivalent to the original non-distributed contrastive loss computation,
without sacrificing any computation accuracy. It is particularly efficient for
large-batch CLIP training. For instance, DisCo-CLIP can enable contrastive
training of a ViT-B/32 model with a batch size of 32K or 196K using 8 or 64
A100 40GB GPUs, compared with the original CLIP solution which requires 128
A100 40GB GPUs to train a ViT-B/32 model with a batch size of 32K. The code
will be released at https://github.com/IDEA-Research/DisCo-CLIP
