  We investigate the concept of Best Approximation for Feedforward Neural
Networks (FNN) and explore their convergence properties through the lens of
Random Projection (RPNNs). RPNNs have predetermined and fixed, once and for
all, internal weights and biases, offering computational efficiency. We
demonstrate that there exists a choice of external weights, for any family of
such RPNNs, with non-polynomial infinitely differentiable activation functions,
that exhibit an exponential convergence rate when approximating any infinitely
differentiable function. For illustration purposes, we test the proposed
RPNN-based function approximation, with parsimoniously chosen basis functions,
across five benchmark function approximation problems. Results show that RPNNs
achieve comparable performance to established methods such as Legendre
Polynomials, highlighting their potential for efficient and accurate function
approximation.
