  There has been a strong demand for algorithms that can execute machine
learning as faster as possible and the speed of deep learning has accelerated
by 30 times only in the past two years. Distributed deep learning using the
large mini-batch is a key technology to address the demand and is a great
challenge as it is difficult to achieve high scalability on large clusters
without compromising accuracy. In this paper, we introduce optimization methods
which we applied to this challenge. We achieved the training time of 74.7
seconds using 2,048 GPUs on ABCI cluster applying these methods. The training
throughput is over 1.73 million images/sec and the top-1 validation accuracy is
75.08%.
