  A wide range of empirical and theoretical works have shown that
overparameterisation can amplify the performance of neural networks. According
to the lottery ticket hypothesis, overparameterised networks have an increased
chance of containing a sub-network that is well-initialised to solve the task
at hand. A more parsimonious approach, inspired by animal learning, consists in
guiding the learner towards solving the task by curating the order of the
examples, i.e. providing a curriculum. However, this learning strategy seems to
be hardly beneficial in deep learning applications. In this work, we undertake
an analytical study that connects curriculum learning and overparameterisation.
In particular, we investigate their interplay in the online learning setting
for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results
show that a high degree of overparameterisation -- while simplifying the
problem -- can limit the benefit from curricula, providing a theoretical
account of the ineffectiveness of curricula in deep learning.
