  Biological nervous systems are created in a fundamentally different way than
current artificial neural networks. Despite its impressive results in a variety
of different domains, deep learning often requires considerable engineering
effort to design high-performing neural architectures. By contrast, biological
nervous systems are grown through a dynamic self-organizing process. In this
paper, we take initial steps toward neural networks that grow through a
developmental process that mirrors key properties of embryonic development in
biological organisms. The growth process is guided by another neural network,
which we call a Neural Developmental Program (NDP) and which operates through
local communication alone. We investigate the role of neural growth on
different machine learning benchmarks and different optimization methods
(evolutionary training, online RL, offline RL, and supervised learning).
Additionally, we highlight future research directions and opportunities enabled
by having self-organization driving the growth of neural networks.
