  We examine the closedness of sets of realized neural networks of a fixed
architecture in Sobolev spaces. For an exactly $m$-times differentiable
activation function $\rho$, we construct a sequence of neural networks
$(\Phi_n)_{n \in \mathbb{N}}$ whose realizations converge in order-$(m-1)$
Sobolev norm to a function that cannot be realized exactly by a neural network.
Thus, sets of realized neural networks are not closed in order-$(m-1)$ Sobolev
spaces $W^{m-1,p}$ for $p \in [1,\infty]$. We further show that these sets are
not closed in $W^{m,p}$ under slightly stronger conditions on the $m$-th
derivative of $\rho$. For a real analytic activation function, we show that
sets of realized neural networks are not closed in $W^{k,p}$ for any $k \in
\mathbb{N}$. The nonclosedness allows for approximation of non-network target
functions with unbounded parameter growth. We partially characterize the rate
of parameter growth for most activation functions by showing that a specific
sequence of realized neural networks can approximate the activation function's
derivative with weights increasing inversely proportional to the $L^p$
approximation error. Finally, we present experimental results showing that
networks are capable of closely approximating non-network target functions with
increasing parameters via training.
