  Recurrent neural networks (RNNs) are brain-inspired models widely used in
machine learning for analyzing sequential data. The present work is a
contribution towards a deeper understanding of how RNNs process input signals
using the response theory from nonequilibrium statistical mechanics. For a
class of continuous-time stochastic RNNs (SRNNs) driven by an input signal, we
derive a Volterra type series representation for their output. This
representation is interpretable and disentangles the input signal from the SRNN
architecture. The kernels of the series are certain recursively defined
correlation functions with respect to the unperturbed dynamics that completely
determine the output. Exploiting connections of this representation and its
implications to rough paths theory, we identify a universal feature -- the
response feature, which turns out to be the signature of tensor product of the
input signal and a natural support basis. In particular, we show that SRNNs,
with only the weights in the readout layer optimized and the weights in the
hidden layer kept fixed and not optimized, can be viewed as kernel machines
operating on a reproducing kernel Hilbert space associated with the response
feature.
