  Deep belief networks (DBNs) are stochastic neural networks that can extract
rich internal representations of the environment from the sensory data. DBNs
had a catalytic effect in triggering the deep learning revolution,
demonstrating for the very first time the feasibility of unsupervised learning
in networks with many layers of hidden neurons. These hierarchical
architectures incorporate plausible biological and cognitive properties, making
them particularly appealing as computational models of human perception and
cognition. However, learning in DBNs is usually carried out in a greedy,
layer-wise fashion, which does not allow to simulate the holistic maturation of
cortical circuits and prevents from modeling cognitive development. Here we
present iDBN, an iterative learning algorithm for DBNs that allows to jointly
update the connection weights across all layers of the model. We evaluate the
proposed iterative algorithm on two different sets of visual stimuli, measuring
the generative capabilities of the learned model and its potential to support
supervised downstream tasks. We also track network development in terms of
graph theoretical properties and investigate the potential extension of iDBN to
continual learning scenarios. DBNs trained using our iterative approach achieve
a final performance comparable to that of the greedy counterparts, at the same
time allowing to accurately analyze the gradual development of internal
representations in the deep network and the progressive improvement in task
performance. Our work paves the way to the use of iDBN for modeling
neurocognitive development.
