  In this paper, we present a local geometric analysis to interpret how deep
feedforward neural networks extract low-dimensional features from
high-dimensional data. Our study shows that, in a local geometric region, the
optimal weight in one layer of the neural network and the optimal feature
generated by the previous layer comprise a low-rank approximation of a matrix
that is determined by the Bayes action of this layer. This result holds (i) for
analyzing both the output layer and the hidden layers of the neural network,
and (ii) for neuron activation functions with non-vanishing gradients. We use
two supervised learning problems to illustrate our results: neural network
based maximum likelihood classification (i.e., softmax regression) and neural
network based minimum mean square estimation. Experimental validation of these
theoretical results will be conducted in our future work.
