  Large language models (LLM) have become a critical component in many
applications of machine learning. However, standard approaches to training LLM
require a large number of tightly interconnected accelerators, with devices
exchanging gradients and other intermediate states at each optimization step.
While it is difficult to build and maintain a single computing cluster hosting
many accelerators, it might be easier to find several computing clusters each
hosting a smaller number of devices. In this work, we propose a distributed
optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
training of language models on islands of devices that are poorly connected.
The approach is a variant of federated averaging, where the number of inner
steps is large, the inner optimizer is AdamW, and the outer optimizer is
Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
workers performs as well as fully synchronous optimization while communicating
500 times less. DiLoCo exhibits great robustness to the data distribution of
each worker. It is also robust to resources becoming unavailable over time, and
vice versa, it can seamlessly leverage resources that become available during
training.
