  In the past decade, we have witnessed the rise of deep learning to dominate
the field of artificial intelligence. Advances in artificial neural networks
alongside corresponding advances in hardware accelerators with large memory
capacity, together with the availability of large datasets enabled
practitioners to train and deploy sophisticated neural network models that
achieve state-of-the-art performance on tasks across several fields spanning
computer vision, natural language processing, and reinforcement learning.
However, as these neural networks become bigger, more complex, and more widely
used, fundamental problems with current deep learning models become more
apparent. State-of-the-art deep learning models are known to suffer from issues
that range from poor robustness, inability to adapt to novel task settings, to
requiring rigid and inflexible configuration assumptions. Collective behavior,
commonly observed in nature, tends to produce systems that are robust,
adaptable, and have less rigid assumptions about the environment configuration.
Collective intelligence, as a field, studies the group intelligence that
emerges from the interactions of many individuals. Within this field, ideas
such as self-organization, emergent behavior, swarm optimization, and cellular
automata were developed to model and explain complex systems. It is therefore
natural to see these ideas incorporated into newer deep learning methods. In
this review, we will provide a historical context of neural network research's
involvement with complex systems, and highlight several active areas in modern
deep learning research that incorporate the principles of collective
intelligence to advance its current capabilities. We hope this review can serve
as a bridge between the complex systems and deep learning communities.
