  Recurrent neural networks (RNNs) are known to be universal approximators of
dynamic systems under fairly mild and general assumptions. However, RNNs
usually suffer from the issues of vanishing and exploding gradients in standard
RNN training. Reservoir computing (RC), a special RNN where the recurrent
weights are randomized and left untrained, has been introduced to overcome
these issues and has demonstrated superior empirical performance especially in
scenarios where training samples are extremely limited. On the other hand, the
theoretical grounding to support this observed performance has yet been fully
developed. In this work, we show that RC can universally approximate a general
linear time-invariant (LTI) system. Specifically, we present a clear signal
processing interpretation of RC and utilize this understanding in the problem
of approximating a generic LTI system. Under this setup, we analytically
characterize the optimum probability density function for configuring (instead
of training and/or randomly generating) the recurrent weights of the underlying
RNN of the RC. Extensive numerical evaluations are provided to validate the
optimality of the derived distribution for configuring the recurrent weights of
the RC to approximate a general LTI system. Our work results in clear signal
processing-based model interpretability of RC and provides theoretical
explanation/justification for the power of randomness in randomly generating
instead of training RC's recurrent weights. Furthermore, it provides a complete
optimum analytical characterization for configuring the untrained recurrent
weights, marking an important step towards explainable machine learning (XML)
to incorporate domain knowledge for efficient learning.
