  The last decade has witnessed a boom of deep learning research and
applications achieving state-of-the-art results in various domains. However,
most advances have been established empirically, and their theoretical analysis
remains lacking. One major issue is that our current interpretation of neural
networks (NNs) as function approximators is too generic to support in-depth
analysis. In this paper, we remedy this by proposing an alternative
interpretation that identifies NNs as chain graphs (CGs) and feed-forward as an
approximate inference procedure. The CG interpretation specifies the nature of
each NN component within the rich theoretical framework of probabilistic
graphical models, while at the same time remains general enough to cover
real-world NNs with arbitrary depth, multi-branching and varied activations, as
well as common structures including convolution / recurrent layers, residual
block and dropout. We demonstrate with concrete examples that the CG
interpretation can provide novel theoretical support and insights for various
NN techniques, as well as derive new deep learning approaches such as the
concept of partially collapsed feed-forward inference. It is thus a promising
framework that deepens our understanding of neural networks and provides a
coherent theoretical formulation for future deep learning research.
