  In training deep learning networks, the optimizer and related learning rate
are often used without much thought or with minimal tuning, even though it is
crucial in ensuring a fast convergence to a good quality minimum of the loss
function that can also generalize well on the test dataset. Drawing inspiration
from the successful application of cyclical learning rate policy for computer
vision related convolutional networks and datasets, we explore how cyclical
learning rate can be applied to train transformer-based neural networks for
neural machine translation. From our carefully designed experiments, we show
that the choice of optimizers and the associated cyclical learning rate policy
can have a significant impact on the performance. In addition, we establish
guidelines when applying cyclical learning rates to neural machine translation
tasks. Thus with our work, we hope to raise awareness of the importance of
selecting the right optimizers and the accompanying learning rate policy, at
the same time, encourage further research into easy-to-use learning rate
policies.
