  The learning speed of feed-forward neural networks is notoriously slow and
has presented a bottleneck in deep learning applications for several decades.
For instance, gradient-based learning algorithms, which are used extensively to
train neural networks, tend to work slowly when all of the network parameters
must be iteratively tuned. To counter this, both researchers and practitioners
have tried introducing randomness to reduce the learning requirement. Based on
the original construction of Igelnik and Pao, single layer neural-networks with
random input-to-hidden layer weights and biases have seen success in practice,
but the necessary theoretical justification is lacking. In this paper, we begin
to fill this theoretical gap. We provide a (corrected) rigorous proof that the
Igelnik and Pao construction is a universal approximator for continuous
functions on compact domains, with approximation error decaying asymptotically
like $O(1/\sqrt{n})$ for the number $n$ of network nodes. We then extend this
result to the non-asymptotic setting, proving that one can achieve any desired
approximation error with high probability provided $n$ is sufficiently large.
We further adapt this randomized neural network architecture to approximate
functions on smooth, compact submanifolds of Euclidean space, providing
theoretical guarantees in both the asymptotic and non-asymptotic forms.
Finally, we illustrate our results on manifolds with numerical experiments.
