  Convex $\ell_1$ regularization using an infinite dictionary of neurons has
been suggested for constructing neural networks with desired approximation
guarantees, but can be affected by an arbitrary amount of over-parametrization.
This can lead to a loss of sparsity and result in networks with too many active
neurons for the given data, in particular if the number of data samples is
large. As a remedy, in this paper, a nonconvex regularization method is
investigated in the context of shallow ReLU networks: We prove that in contrast
to the convex approach, any resulting (locally optimal) network is finite even
in the presence of infinite data (i.e., if the data distribution is known and
the limiting case of infinite samples is considered). Moreover, we show that
approximation guarantees and existing bounds on the network size for finite
data are maintained.
