  We study the theory of neural network (NN) from the lens of classical
nonparametric regression problems with a focus on NN's ability to adaptively
estimate functions with heterogeneous smoothness -- a property of functions in
Besov or Bounded Variation (BV) classes. Existing work on this problem requires
tuning the NN architecture based on the function spaces and sample size. We
consider a "Parallel NN" variant of deep ReLU networks and show that the
standard $\ell_2$ regularization is equivalent to promoting the
$\ell_p$-sparsity ($0<p<1$) in the coefficient vector of an end-to-end learned
function bases, i.e., a dictionary. Using this equivalence, we further
establish that by tuning only the regularization factor, such parallel NN
achieves an estimation error arbitrarily close to the minimax rates for both
the Besov and BV classes. Notably, it gets exponentially closer to minimax
optimal as the NN gets deeper. Our research sheds new lights on why depth
matters and how NNs are more powerful than kernel methods.
