  Increasing the capacity of recurrent neural networks (RNN) usually involves
augmenting the size of the hidden layer, with significant increase of
computational cost. Recurrent neural tensor networks (RNTN) increase capacity
using distinct hidden layer weights for each word, but with greater costs in
memory usage. In this paper, we introduce restricted recurrent neural tensor
networks (r-RNTN) which reserve distinct hidden layer weights for frequent
vocabulary words while sharing a single set of weights for infrequent words.
Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve
language model performance over RNNs using only a small fraction of the
parameters of unrestricted RNTNs. These results hold for r-RNTNs using Gated
Recurrent Units and Long Short-Term Memory.
