  Training neural networks with many processors can reduce time-to-solution;
however, it is challenging to maintain convergence and efficiency at large
scales. The Kronecker-factored Approximate Curvature (K-FAC) was recently
proposed as an approximation of the Fisher Information Matrix that can be used
in natural gradient optimizers. We investigate here a scalable K-FAC design and
its applicability in convolutional neural network (CNN) training at scale. We
study optimization techniques such as layer-wise distribution strategies,
inverse-free second-order gradient evaluation, and dynamic K-FAC update
decoupling to reduce training time while preserving convergence. We use
residual neural networks (ResNet) applied to the CIFAR-10 and ImageNet-1k
datasets to evaluate the correctness and scalability of our K-FAC gradient
preconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributed
K-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less time
than does the classic stochastic gradient descent (SGD) optimizer across scales
on a GPU cluster.
