  We explore the phase diagram of approximation rates for deep neural networks
and prove several new theoretical results. In particular, we generalize the
existing result on the existence of deep discontinuous phase in ReLU networks
to functional classes of arbitrary positive smoothness, and identify the
boundary between the feasible and infeasible rates. Moreover, we show that all
networks with a piecewise polynomial activation function have the same phase
diagram. Next, we demonstrate that standard fully-connected architectures with
a fixed width independent of smoothness can adapt to smoothness and achieve
almost optimal rates. Finally, we consider deep networks with periodic
activations ("deep Fourier expansion") and prove that they have very fast,
nearly exponential approximation rates, thanks to the emerging capability of
the network to implement efficient lookup operations.
