  Learning rate scheduler has been a critical issue in the deep neural network
training. Several schedulers and methods have been proposed, including step
decay scheduler, adaptive method, cosine scheduler and cyclical scheduler. This
paper proposes a new scheduling method, named hyperbolic-tangent decay (HTD).
We run experiments on several benchmarks such as: ResNet, Wide ResNet and
DenseNet for CIFAR-10 and CIFAR-100 datasets, LSTM for PAMAP2 dataset, ResNet
on ImageNet and Fashion-MNIST datasets. In our experiments, HTD outperforms
step decay and cosine scheduler in nearly all cases, while requiring less
hyperparameters than step decay, and more flexible than cosine scheduler. Code
is available at https://github.com/BIGBALLON/HTD.
