  Local SGD is a communication-efficient variant of SGD for large-scale
training, where multiple GPUs perform SGD independently and average the model
parameters periodically. It has been recently observed that Local SGD can not
only achieve the design goal of reducing the communication overhead but also
lead to higher test accuracy than the corresponding SGD baseline (Lin et al.,
2020b), though the training regimes for this to happen are still in debate
(Ortiz et al., 2021). This paper aims to understand why (and when) Local SGD
generalizes better based on Stochastic Differential Equation (SDE)
approximation. The main contributions of this paper include (i) the derivation
of an SDE that captures the long-term behavior of Local SGD in the small
learning rate regime, showing how noise drives the iterate to drift and diffuse
after it has reached close to the manifold of local minima, (ii) a comparison
between the SDEs of Local SGD and SGD, showing that Local SGD induces a
stronger drift term that can result in a stronger effect of regularization,
e.g., a faster reduction of sharpness, and (iii) empirical evidence validating
that having a small learning rate and long enough training time enables the
generalization improvement over SGD but removing either of the two conditions
leads to no improvement.
