  A proper understanding of the striking generalization abilities of deep
neural networks presents an enduring puzzle. Recently, there has been a growing
body of numerically-grounded theoretical work that has contributed important
insights to the theory of learning in deep neural nets. There has also been a
recent interest in extending these analyses to understanding how multitask
learning can further improve the generalization capacity of deep neural nets.
These studies deal almost exclusively with regression tasks which are amenable
to existing analytical techniques. We develop an analytic theory of the
nonlinear dynamics of generalization of deep neural networks trained to solve
classification tasks using softmax outputs and cross-entropy loss, addressing
both single task and multitask settings. We do so by adapting techniques from
the statistical physics of disordered systems, accounting for both finite size
datasets and correlated outputs induced by the training dynamics. We discuss
the validity of our theoretical results in comparison to a comprehensive suite
of numerical experiments. Our analysis provides theoretical support for the
intuition that the performance of multitask learning is determined by the
noisiness of the tasks and how well their input features align with each other.
Highly related, clean tasks benefit each other, whereas unrelated, clean tasks
can be detrimental to individual task performance.
