  The choice of hyper-parameters affects the performance of neural models.
While much previous research (Sutskever et al., 2013; Duchi et al., 2011;
Kingma and Ba, 2015) focuses on accelerating convergence and reducing the
effects of the learning rate, comparatively few papers concentrate on the
effect of batch size. In this paper, we analyze how increasing batch size
affects gradient direction, and propose to evaluate the stability of gradients
with their angle change. Based on our observations, the angle change of
gradient direction first tends to stabilize (i.e. gradually decrease) while
accumulating mini-batches, and then starts to fluctuate. We propose to
automatically and dynamically determine batch sizes by accumulating gradients
of mini-batches and performing an optimization step at just the time when the
direction of gradients starts to fluctuate. To improve the efficiency of our
approach for large models, we propose a sampling approach to select gradients
of parameters sensitive to the batch size. Our approach dynamically determines
proper and efficient batch sizes during training. In our experiments on the WMT
14 English to German and English to French tasks, our approach improves the
Transformer with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively.
