  It is common in deep learning to warm up the learning rate $\eta$, often by a
linear schedule between $\eta_{\text{init}} = 0$ and a predetermined target
$\eta_{\text{trgt}}$. In this paper, we show through systematic experiments
using SGD and Adam that the overwhelming benefit of warmup arises from allowing
the network to tolerate larger $\eta_{\text{trgt}}$ by forcing the network to
more well-conditioned areas of the loss landscape. The ability to handle larger
$\eta_{\text{trgt}}$ makes hyperparameter tuning more robust while improving
the final performance. We uncover different regimes of operation during the
warmup period, depending on whether training starts off in a progressive
sharpening or sharpness reduction phase, which in turn depends on the
initialization and parameterization. Using these insights, we show how
$\eta_{\text{init}}$ can be properly chosen by utilizing the loss catapult
mechanism, which saves on the number of warmup steps, in some cases completely
eliminating the need for warmup. We also suggest an initialization for the
variance in Adam which provides benefits similar to warmup.
