  The Recurrent Neural Networks and their variants have shown promising
performances in sequence modeling tasks such as Natural Language Processing.
These models, however, turn out to be impractical and difficult to train when
exposed to very high-dimensional inputs due to the large input-to-hidden weight
matrix. This may have prevented RNNs' large-scale application in tasks that
involve very high input dimensions such as video modeling; current approaches
reduce the input dimensions using various feature extractors. To address this
challenge, we propose a new, more general and efficient approach by factorizing
the input-to-hidden weight matrix using Tensor-Train decomposition which is
trained simultaneously with the weights themselves. We test our model on
classification tasks using multiple real-world video datasets and achieve
competitive performances with state-of-the-art models, even though our model
architecture is orders of magnitude less complex. We believe that the proposed
approach provides a novel and fundamental building block for modeling
high-dimensional sequential data with RNN architectures and opens up many
possibilities to transfer the expressive and advanced architectures from other
domains such as NLP to modeling high-dimensional sequential data.
