  Background: Deep learning models are typically trained using stochastic
gradient descent or one of its variants. These methods update the weights using
their gradient, estimated from a small fraction of the training data. It has
been observed that when using large batch sizes there is a persistent
degradation in generalization performance - known as the "generalization gap"
phenomena. Identifying the origin of this gap and closing it had remained an
open problem.
  Contributions: We examine the initial high learning rate training phase. We
find that the weight distance from its initialization grows logarithmically
with the number of weight updates. We therefore propose a "random walk on
random landscape" statistical model which is known to exhibit similar
"ultra-slow" diffusion behavior. Following this hypothesis we conducted
experiments to show empirically that the "generalization gap" stems from the
relatively small number of updates rather than the batch size, and can be
completely eliminated by adapting the training regime used. We further
investigate different techniques to train models in the large-batch regime and
present a novel algorithm named "Ghost Batch Normalization" which enables
significant decrease in the generalization gap without increasing the number of
updates. To validate our findings we conduct several additional experiments on
MNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices
and beliefs concerning training of deep models and suggest they may not be
optimal to achieve good generalization.
