  We establish $ L^{\infty} $ and $ L^2 $ error bounds for functions of many
variables that are approximated by linear combinations of ReLU (rectified
linear unit) and squared ReLU ridge functions with $ \ell^1 $ and $ \ell^0 $
controls on their inner and outer parameters. With the squared ReLU ridge
function, we show that the $ L^2 $ approximation error is inversely
proportional to the inner layer $ \ell^0 $ sparsity and it need only be
sublinear in the outer layer $ \ell^0 $ sparsity. Our constructions are
obtained using a variant of the Jones-Barron probabilistic method, which can be
interpreted as either stratified sampling with proportionate allocation or
two-stage cluster sampling. We also provide companion error lower bounds that
reveal near optimality of our constructions. Despite the sparsity assumptions,
we showcase the richness and flexibility of these ridge combinations by
defining a large family of functions, in terms of certain spectral conditions,
that are particularly well approximated by them.
