  Despite the development of numerous adaptive optimizers, tuning the learning
rate of stochastic gradient methods remains a major roadblock to obtaining good
practical performance in machine learning. Rather than changing the learning
rate at each iteration, we propose an approach that automates the most common
hand-tuning heuristic: use a constant learning rate until "progress stops,"
then drop. We design an explicit statistical test that determines when the
dynamics of stochastic gradient descent reach a stationary distribution. This
test can be performed easily during training, and when it fires, we decrease
the learning rate by a constant multiplicative factor. Our experiments on
several deep learning tasks demonstrate that this statistical adaptive
stochastic approximation (SASA) method can automatically find good learning
rate schedules and match the performance of hand-tuned methods using default
settings of its parameters. The statistical testing helps to control the
variance of this procedure and improves its robustness.
