  A neural network is locally specialized to the extent that parts of its
computational graph (i.e. structure) can be abstractly represented as
performing some comprehensible sub-task relevant to the overall task (i.e.
functionality). Are modern deep neural networks locally specialized? How can
this be quantified? In this paper, we consider the problem of taking a neural
network whose neurons are partitioned into clusters, and quantifying how
functionally specialized the clusters are. We propose two proxies for this:
importance, which reflects how crucial sets of neurons are to network
performance; and coherence, which reflects how consistently their neurons
associate with features of the inputs. To measure these proxies, we develop a
set of statistical methods based on techniques conventionally used to interpret
individual neurons. We apply the proxies to partitionings generated by
spectrally clustering a graph representation of the network's neurons with
edges determined either by network weights or correlations of activations. We
show that these partitionings, even ones based only on weights (i.e. strictly
from non-runtime analysis), reveal groups of neurons that are important and
coherent. These results suggest that graph-based partitioning can reveal local
specialization and that statistical methods can be used to automatedly screen
for sets of neurons that can be understood abstractly.
