  We provide several new depth-based separation results for feed-forward neural
networks, proving that various types of simple and natural functions can be
better approximated using deeper networks than shallower ones, even if the
shallower networks are much larger. This includes indicators of balls and
ellipses; non-linear functions which are radial with respect to the $L_1$ norm;
and smooth non-linear functions. We also show that these gaps can be observed
experimentally: Increasing the depth indeed allows better learning than
increasing width, when training neural networks to learn an indicator of a unit
ball.
