  Most curriculum learning methods require an approach to sort the data samples
by difficulty, which is often cumbersome to perform. In this work, we propose a
novel curriculum learning approach termed Learning Rate Curriculum (LeRaC),
which leverages the use of a different learning rate for each layer of a neural
network to create a data-agnostic curriculum during the initial training
epochs. More specifically, LeRaC assigns higher learning rates to neural layers
closer to the input, gradually decreasing the learning rates as the layers are
placed farther away from the input. The learning rates increase at various
paces during the first training iterations, until they all reach the same
value. From this point on, the neural model is trained as usual. This creates a
model-level curriculum learning strategy that does not require sorting the
examples by difficulty and is compatible with any neural network, generating
higher performance levels regardless of the architecture. We conduct
comprehensive experiments on 12 data sets from the computer vision (CIFAR-10,
CIFAR-100, Tiny ImageNet, ImageNet-200, Food-101, UTKFace, PASCAL VOC),
language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering
various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121, YOLOv5),
recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures. We compare
our approach with the conventional training regime, as well as with Curriculum
by Smoothing (CBS), a state-of-the-art data-agnostic curriculum learning
approach. Unlike CBS, our performance improvements over the standard training
regime are consistent across all data sets and models. Furthermore, we
significantly surpass CBS in terms of training time (there is no additional
cost over the standard training regime for LeRaC). Our code is freely available
at: https://github.com/CroitoruAlin/LeRaC.
