  We study the expressivity of deep neural networks. Measuring a network's
complexity by its number of connections or by its number of neurons, we
consider the class of functions for which the error of best approximation with
networks of a given complexity decays at a certain rate when increasing the
complexity budget. Using results from classical approximation theory, we show
that this class can be endowed with a (quasi)-norm that makes it a linear
function space, called approximation space. We establish that allowing the
networks to have certain types of "skip connections" does not change the
resulting approximation spaces. We also discuss the role of the network's
nonlinearity (also known as activation function) on the resulting spaces, as
well as the role of depth. For the popular ReLU nonlinearity and its powers, we
relate the newly constructed spaces to classical Besov spaces. The established
embeddings highlight that some functions of very low Besov smoothness can
nevertheless be well approximated by neural networks, if these networks are
sufficiently deep.
