  During language acquisition, children follow a typical sequence of learning
stages, whereby they first learn to categorize phonemes before they develop
their lexicon and eventually master increasingly complex syntactic structures.
However, the computational principles that lead to this learning trajectory
remain largely unknown. To investigate this, we here compare the learning
trajectories of deep language models to those of children. Specifically, we
test whether, during its training, GPT-2 exhibits stages of language
acquisition comparable to those observed in children aged between 18 months and
6 years. For this, we train 48 GPT-2 models from scratch and evaluate their
syntactic and semantic abilities at each training step, using 96 probes curated
from the BLiMP, Zorro and BIG-Bench benchmarks. We then compare these
evaluations with the behavior of 54 children during language production. Our
analyses reveal three main findings. First, similarly to children, the language
models tend to learn linguistic skills in a systematic order. Second, this
learning scheme is parallel: the language tasks that are learned last improve
from the very first training steps. Third, some - but not all - learning stages
are shared between children and these language models. Overall, these results
shed new light on the principles of language acquisition, and highlight
important divergences in how humans and modern algorithms learn to process
natural language.
