  Recurrent neural networks (RNNs) are powerful tools for sequential modeling,
but typically require significant overparameterization and regularization to
achieve optimal performance. This leads to difficulties in the deployment of
large RNNs in resource-limited settings, while also introducing complications
in hyperparameter selection and training. To address these issues, we introduce
a "fully tensorized" RNN architecture which jointly encodes the separate weight
matrices within each recurrent cell using a lightweight tensor-train (TT)
factorization. This approach represents a novel form of weight sharing which
reduces model size by several orders of magnitude, while still maintaining
similar or better performance compared to standard RNNs. Experiments on image
classification and speaker verification tasks demonstrate further benefits for
reducing inference times and stabilizing model training and hyperparameter
selection.
