  Deep learning of artificial neural networks (ANNs) is creating highly
functional processes that are, unfortunately, nearly as hard to interpret as
their biological counterparts. Identification of functional modules in natural
brains plays an important role in cognitive and neuroscience alike, and can be
carried out using a wide range of technologies such as fMRI, EEG/ERP, MEG, or
calcium imaging. However, we do not have such robust methods at our disposal
when it comes to understanding functional modules in artificial neural
networks. Ideally, understanding which parts of an artificial neural network
perform what function might help us to address a number of vexing problems in
ANN research, such as catastrophic forgetting and overfitting. Furthermore,
revealing a network's modularity could improve our trust in them by making
these black boxes more transparent. Here, we introduce a new
information-theoretic concept that proves useful in understanding and analyzing
a network's functional modularity: the relay information $I_R$. The relay
information measures how much information groups of neurons that participate in
a particular function (modules) relay from inputs to outputs. Combined with a
greedy search algorithm, relay information can be used to identify
computational modules in neural networks. We also show that the functionality
of modules correlates with the amount of relay information they carry.
