  Recent results in nonparametric regression show that deep learning, i.e.,
neural network estimates with many hidden layers, are able to circumvent the
so-called curse of dimensionality in case that suitable restrictions on the
structure of the regression function hold. One key feature of the neural
networks used in these results is that their network architecture has a further
constraint, namely the network sparsity. In this paper we show that we can get
similar results also for least squares estimates based on simple fully
connected neural networks with ReLU activation functions. Here either the
number of neurons per hidden layer is fixed and the number of hidden layers
tends to infinity suitably fast for sample size tending to infinity, or the
number of hidden layers is bounded by some logarithmic factor in the sample
size and the number of neurons per hidden layer tends to infinity suitably fast
for sample size tending to infinity. The proof is based on new approximation
results concerning deep neural networks.
