  We study the efficacy and efficiency of deep generative networks for
approximating probability distributions. We prove that neural networks can
transform a low-dimensional source distribution to a distribution that is
arbitrarily close to a high-dimensional target distribution, when the closeness
are measured by Wasserstein distances and maximum mean discrepancy. Upper
bounds of the approximation error are obtained in terms of the width and depth
of neural network. Furthermore, it is shown that the approximation error in
Wasserstein distance grows at most linearly on the ambient dimension and that
the approximation order only depends on the intrinsic dimension of the target
distribution. On the contrary, when $f$-divergences are used as metrics of
distributions, the approximation property is different. We show that in order
to approximate the target distribution in $f$-divergences, the dimension of the
source distribution cannot be smaller than the intrinsic dimension of the
target distribution.
