  Large-scale distributed training of deep neural networks results in models
with worse generalization performance as a result of the increase in the
effective mini-batch size. Previous approaches attempt to address this problem
by varying the learning rate and batch size over epochs and layers, or ad hoc
modifications of batch normalization. We propose Scalable and Practical Natural
Gradient Descent (SP-NGD), a principled approach for training models that
allows them to attain similar generalization performance to models trained with
first-order optimization methods, but with accelerated convergence.
Furthermore, SP-NGD scales to large mini-batch sizes with a negligible
computational overhead as compared to first-order methods. We evaluated SP-NGD
on a benchmark task where highly optimized first-order methods are available as
references: training a ResNet-50 model for image classification on ImageNet. We
demonstrate convergence to a top-1 validation accuracy of 75.4% in 5.5 minutes
using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of
74.9% with an extremely large mini-batch size of 131,072 in 873 steps of
SP-NGD.
