  The study of deep neural networks (DNNs) in the infinite-width limit, via the
so-called neural tangent kernel (NTK) approach, has provided new insights into
the dynamics of learning, generalization, and the impact of initialization. One
key DNN architecture remains to be kernelized, namely, the recurrent neural
network (RNN). In this paper we introduce and study the Recurrent Neural
Tangent Kernel (RNTK), which provides new insights into the behavior of
overparametrized RNNs. A key property of the RNTK should greatly benefit
practitioners is its ability to compare inputs of different length. To this
end, we characterize how the RNTK weights different time steps to form its
output under different initialization parameters and nonlinearity choices. A
synthetic and 56 real-world data experiments demonstrate that the RNTK offers
significant performance gains over other kernels, including standard NTKs,
across a wide array of data sets.
