  We study the properties of nonparametric least squares regression using deep
neural networks. We derive non-asymptotic upper bounds for the prediction error
of the empirical risk minimizer of feedforward deep neural regression. Our
error bounds achieve minimax optimal rate and significantly improve over the
existing ones in the sense that they depend polynomially on the dimension of
the predictor, instead of exponentially on dimension. We show that the neural
regression estimator can circumvent the curse of dimensionality under the
assumption that the predictor is supported on an approximate low-dimensional
manifold or a set with low Minkowski dimension. We also establish the optimal
convergence rate under the exact manifold support assumption. We investigate
how the prediction error of the neural regression estimator depends on the
structure of neural networks and propose a notion of network relative
efficiency between two types of neural networks, which provides a quantitative
measure for evaluating the relative merits of different network structures. To
establish these results, we derive a novel approximation error bound for the
H\"older smooth functions with a positive smoothness index using ReLU activated
neural networks, which may be of independent interest. Our results are derived
under weaker assumptions on the data distribution and the neural network
structure than those in the existing literature.
