  The learned weights of a neural network are often considered devoid of
scrutable internal structure. To discern structure in these weights, we
introduce a measurable notion of modularity for multi-layer perceptrons (MLPs),
and investigate the modular structure of MLPs trained on datasets of small
images. Our notion of modularity comes from the graph clustering literature: a
"module" is a set of neurons with strong internal connectivity but weak
external connectivity. We find that training and weight pruning produces MLPs
that are more modular than randomly initialized ones, and often significantly
more modular than random MLPs with the same (sparse) distribution of weights.
Interestingly, they are much more modular when trained with dropout. We also
present exploratory analyses of the importance of different modules for
performance and how modules depend on each other. Understanding the modular
structure of neural networks, when such structure exists, will hopefully render
their inner workings more interpretable to engineers. Note that this paper has
been superceded by "Clusterability in Neural Networks", arxiv:2103.03386 and
"Quantifying Local Specialization in Deep Neural Networks", arxiv:2110.08058!
