  Along with the rapid development of deep learning in practice, the
theoretical explanations for its success become urgent. Generalization and
expressivity are two widely used measurements to quantify theoretical behaviors
of deep learning. The expressivity focuses on finding functions expressible by
deep nets but cannot be approximated by shallow nets with the similar number of
neurons. It usually implies the large capacity. The generalization aims at
deriving fast learning rate for deep nets. It usually requires small capacity
to reduce the variance. Different from previous studies on deep learning,
pursuing either expressivity or generalization, we take both factors into
account to explore the theoretical advantages of deep nets. For this purpose,
we construct a deep net with two hidden layers possessing excellent
expressivity in terms of localized and sparse approximation. Then, utilizing
the well known covering number to measure the capacity, we find that deep nets
possess excellent expressive power (measured by localized and sparse
approximation) without enlarging the capacity of shallow nets. As a
consequence, we derive near optimal learning rates for implementing empirical
risk minimization (ERM) on the constructed deep nets. These results
theoretically exhibit the advantage of deep nets from learning theory
viewpoints.
