  The past decade has witnessed a successful application of deep learning to
solving many challenging problems in machine learning and artificial
intelligence. However, the loss functions of deep neural networks (especially
nonlinear networks) are still far from being well understood from a theoretical
aspect. In this paper, we enrich the current understanding of the landscape of
the square loss functions for three types of neural networks. Specifically,
when the parameter matrices are square, we provide an explicit characterization
of the global minimizers for linear networks, linear residual networks, and
nonlinear networks with one hidden layer. Then, we establish two quadratic
types of landscape properties for the square loss of these neural networks,
i.e., the gradient dominance condition within the neighborhood of their full
rank global minimizers, and the regularity condition along certain directions
and within the neighborhood of their global minimizers. These two landscape
properties are desirable for the optimization around the global minimizers of
the loss function for these neural networks.
