  Understanding the inner workings of deep neural networks (DNNs) is essential
to provide trustworthy artificial intelligence techniques for practical
applications. Existing studies typically involve linking semantic concepts to
units or layers of DNNs, but fail to explain the inference process. In this
paper, we introduce neural architecture disentanglement (NAD) to fill the gap.
Specifically, NAD learns to disentangle a pre-trained DNN into
sub-architectures according to independent tasks, forming information flows
that describe the inference processes. We investigate whether, where, and how
the disentanglement occurs through experiments conducted with handcrafted and
automatically-searched network architectures, on both object-based and
scene-based datasets. Based on the experimental results, we present three new
findings that provide fresh insights into the inner logic of DNNs. First, DNNs
can be divided into sub-architectures for independent tasks. Second, deeper
layers do not always correspond to higher semantics. Third, the connection type
in a DNN affects how the information flows across layers, leading to different
disentanglement behaviors. With NAD, we further explain why DNNs sometimes give
wrong predictions. Experimental results show that misclassified images have a
high probability of being assigned to task sub-architectures similar to the
correct ones. Code will be available at: https://github.com/hujiecpp/NAD.
