  We show that the ability of a neural network to integrate information from
diverse sources hinges critically on being exposed to properly correlated
signals during the early phases of training. Interfering with the learning
process during this initial stage can permanently impair the development of a
skill, both in artificial and biological systems where the phenomenon is known
as a critical learning period. We show that critical periods arise from the
complex and unstable early transient dynamics, which are decisive of final
performance of the trained system and their learned representations. This
evidence challenges the view, engendered by analysis of wide and shallow
networks, that early learning dynamics of neural networks are simple, akin to
those of a linear model. Indeed, we show that even deep linear networks exhibit
critical learning periods for multi-source integration, while shallow networks
do not. To better understand how the internal representations change according
to disturbances or sensory deficits, we introduce a new measure of source
sensitivity, which allows us to track the inhibition and integration of sources
during training. Our analysis of inhibition suggests cross-source
reconstruction as a natural auxiliary training objective, and indeed we show
that architectures trained with cross-sensor reconstruction objectives are
remarkably more resilient to critical periods. Our findings suggest that the
recent success in self-supervised multi-modal training compared to previous
supervised efforts may be in part due to more robust learning dynamics and not
solely due to better architectures and/or more data.
