  Artificial neural networks (ANNs) have achieved significant success in
tackling classical and modern machine learning problems. As learning problems
grow in scale and complexity, and expand into multi-disciplinary territory, a
more modular approach for scaling ANNs will be needed. Modular neural networks
(MNNs) are neural networks that embody the concepts and principles of
modularity. MNNs adopt a large number of different techniques for achieving
modularization. Previous surveys of modularization techniques are relatively
scarce in their systematic analysis of MNNs, focusing mostly on empirical
comparisons and lacking an extensive taxonomical framework. In this review, we
aim to establish a solid taxonomy that captures the essential properties and
relationships of the different variants of MNNs. Based on an investigation of
the different levels at which modularization techniques act, we attempt to
provide a universal and systematic framework for theorists studying MNNs, also
trying along the way to emphasise the strengths and weaknesses of different
modularization approaches in order to highlight good practices for neural
network practitioners.
