  Understanding the loss surface of a neural network is fundamentally important
to the understanding of deep learning. This paper presents how piecewise linear
activation functions substantially shape the loss surfaces of neural networks.
We first prove that {\it the loss surfaces of many neural networks have
infinite spurious local minima} which are defined as the local minima with
higher empirical risks than the global minima. Our result demonstrates that the
networks with piecewise linear activations possess substantial differences to
the well-studied linear neural networks. This result holds for any neural
network with arbitrary depth and arbitrary piecewise linear activation
functions (excluding linear functions) under most loss functions in practice.
Essentially, the underlying assumptions are consistent with most practical
circumstances where the output layer is narrower than any hidden layer. In
addition, the loss surface of a neural network with piecewise linear
activations is partitioned into multiple smooth and multilinear cells by
nondifferentiable boundaries. The constructed spurious local minima are
concentrated in one cell as a valley: they are connected with each other by a
continuous path, on which empirical risk is invariant. Further for
one-hidden-layer networks, we prove that all local minima in a cell constitute
an equivalence class; they are concentrated in a valley; and they are all
global minima in the cell.
