  Current state-of-the-art NLP systems use large neural networks that require
lots of computational resources for training. Inspired by human knowledge
acquisition, researchers have proposed curriculum learning, - sequencing of
tasks (task-based curricula) or ordering and sampling of the datasets
(data-based curricula) that facilitate training. This work investigates the
benefits of data-based curriculum learning for large modern language models
such as BERT and T5. We experiment with various curricula based on a range of
complexity measures and different sampling strategies. Extensive experiments on
different NLP tasks show that curricula based on various complexity measures
rarely has any benefits while random sampling performs either as well or better
than curricula.
