  Unraveling the emergence of collective learning in systems of coupled
artificial neural networks points to broader implications for machine learning,
neuroscience, and society. Here we introduce a minimal model that condenses
several recent decentralized algorithms by considering a competition between
two terms: the local learning dynamics in the parameters of each neural network
unit, and a diffusive coupling among units that tends to homogenize the
parameters of the ensemble. We derive an effective theory for linear networks
to show that the coarse-grained behavior of our system is equivalent to a
deformed Ginzburg-Landau model with quenched disorder. This framework predicts
depth-dependent disorder-order-disorder phase transitions in the parameters'
solutions that reveal a depth-delayed onset of a collective learning phase and
a low-rank microscopic learning path. We validate the theory in coupled
ensembles of realistic neural networks trained on the MNIST dataset under
privacy constraints. Interestingly, experiments confirm that individual
networks -- trained on private data -- can fully generalize to unseen data
classes when the collective learning phase emerges. Our work establishes the
physics of collective learning and contributes to the mechanistic
interpretability of deep learning in decentralized settings.
