  Since statistical guarantees for neural networks are usually restricted to
global optima of intricate objective functions, it is not clear whether these
theories really explain the performances of actual outputs of neural-network
pipelines. The goal of this paper is, therefore, to bring statistical theory
closer to practice. We develop statistical guarantees for simple neural
networks that coincide up to logarithmic factors with the global optima but
apply to stationary points and the points nearby. These results support the
common notion that neural networks do not necessarily need to be optimized
globally from a mathematical perspective. More generally, despite being limited
to simple neural networks for now, our theories make a step forward in
describing the practical properties of neural networks in mathematical terms.
