  This paper reviews recent studies in understanding neural-network
representations and learning neural networks with interpretable/disentangled
middle-layer representations. Although deep neural networks have exhibited
superior performance in various tasks, the interpretability is always the
Achilles' heel of deep neural networks. At present, deep neural networks obtain
high discrimination power at the cost of low interpretability of their
black-box representations. We believe that high model interpretability may help
people to break several bottlenecks of deep learning, e.g., learning from very
few annotations, learning via human-computer communications at the semantic
level, and semantically debugging network representations. We focus on
convolutional neural networks (CNNs), and we revisit the visualization of CNN
representations, methods of diagnosing representations of pre-trained CNNs,
approaches for disentangling pre-trained CNN representations, learning of CNNs
with disentangled representations, and middle-to-end learning based on model
interpretability. Finally, we discuss prospective trends in explainable
artificial intelligence.
