  In current deep learning tasks, Adam style optimizers such as Adam, Adagrad,
RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style
optimizers. These optimizers typically update model parameters using the sign
of gradients, resulting in more stable convergence curves. The learning rate
and the batch size are the most critical hyperparameters for optimizers, which
require careful tuning to enable effective convergence. Previous research has
shown that the optimal learning rate increases linearly or follows similar
rules with batch size for SGD style optimizers. However, this conclusion is not
applicable to Adam style optimizers. In this paper, we elucidate the connection
between optimal learning rates and batch sizes for Adam style optimizers
through both theoretical analysis and extensive experiments. First, we raise
the scaling law between batch sizes and optimal learning rates in the sign of
gradient case, in which we prove that the optimal learning rate first rises and
then falls as the batch size increases. Moreover, the peak value of the surge
will gradually move toward the larger batch size as training progresses.
Second, we conducted experiments on various CV and NLP tasks and verified the
correctness of the scaling law.
