  This paper considers the power of deep neural networks (deep nets for short)
in realizing data features. Based on refined covering number estimates, we find
that, to realize some complex data features, deep nets can improve the
performances of shallow neural networks (shallow nets for short) without
requiring additional capacity costs. This verifies the advantage of deep nets
in realizing complex features. On the other hand, to realize some simple data
feature like the smoothness, we prove that, up to a logarithmic factor, the
approximation rate of deep nets is asymptotically identical to that of shallow
nets, provided that the depth is fixed. This exhibits a limitation of deep nets
in realizing simple features.
