  Neural networks (NNs) whose subnetworks implement reusable functions are
expected to offer numerous advantages, including compositionality through
efficient recombination of functional building blocks, interpretability,
preventing catastrophic interference, etc. Understanding if and how NNs are
modular could provide insights into how to improve them. Current inspection
methods, however, fail to link modules to their functionality. In this paper,
we present a novel method based on learning binary weight masks to identify
individual weights and subnets responsible for specific functions. Using this
powerful tool, we contribute an extensive study of emerging modularity in NNs
that covers several standard architectures and datasets. We demonstrate how
common NNs fail to reuse submodules and offer new insights into the related
issue of systematic generalization on language tasks.
