  We investigate the expressive power of depth-2 bandlimited random neural
networks. A random net is a neural network where the hidden layer parameters
are frozen with random assignment, and only the output layer parameters are
trained by loss minimization. Using random weights for a hidden layer is an
effective method to avoid non-convex optimization in standard gradient descent
learning. It has also been adopted in recent deep learning theories. Despite
the well-known fact that a neural network is a universal approximator, in this
study, we mathematically show that when hidden parameters are distributed in a
bounded domain, the network may not achieve zero approximation error. In
particular, we derive a new nontrivial approximation error lower bound. The
proof utilizes the technique of ridgelet analysis, a harmonic analysis method
designed for neural networks. This method is inspired by fundamental principles
in classical signal processing, specifically the idea that signals with limited
bandwidth may not always be able to perfectly recreate the original signal. We
corroborate our theoretical results with various simulation studies, and
generally, two main take-home messages are offered: (i) Not any distribution
for selecting random weights is feasible to build a universal approximator;
(ii) A suitable assignment of random weights exists but to some degree is
associated with the complexity of the target function.
