  Deep neural networks learn increasingly complex functions over the course of
training. Here, we show both empirically and theoretically that learning of the
target function is preceded by an early phase in which networks learn the
optimal constant solution (OCS) - that is, initial model responses mirror the
distribution of target labels, while entirely ignoring information provided in
the input. Using a hierarchical category learning task, we derive exact
solutions for learning dynamics in deep linear networks trained with bias
terms. Even when initialized to zero, this simple architectural feature induces
substantial changes in early dynamics. We identify hallmarks of this early OCS
phase and illustrate how these signatures are observed in deep linear networks
and larger, more complex (and nonlinear) convolutional neural networks solving
a hierarchical learning task based on MNIST and CIFAR10. We explain these
observations by proving that deep linear networks necessarily learn the OCS
during early learning. To further probe the generality of our results, we train
human learners over the course of three days on the category learning task. We
then identify qualitative signatures of this early OCS phase in terms of the
dynamics of true negative (correct-rejection) rates. Surprisingly, we find the
same early reliance on the OCS in the behaviour of human learners. Finally, we
show that learning of the OCS can emerge even in the absence of bias terms and
is equivalently driven by generic correlations in the input data. Overall, our
work suggests the OCS as a universal learning principle in supervised,
error-corrective learning, and the mechanistic reasons for its prevalence.
