  The embedding layers transforming input words into real vectors are the key
components of deep neural networks used in natural language processing.
However, when the vocabulary is large, the corresponding weight matrices can be
enormous, which precludes their deployment in a limited resource setting. We
introduce a novel way of parametrizing embedding layers based on the Tensor
Train (TT) decomposition, which allows compressing the model significantly at
the cost of a negligible drop or even a slight gain in performance. We evaluate
our method on a wide range of benchmarks in natural language processing and
analyze the trade-off between performance and compression ratios for a wide
range of architectures, from MLPs to LSTMs and Transformers.
