  Language models must capture statistical dependencies between words at
timescales ranging from very short to very long. Earlier work has demonstrated
that dependencies in natural language tend to decay with distance between words
according to a power law. However, it is unclear how this knowledge can be used
for analyzing or designing neural network language models. In this work, we
derived a theory for how the memory gating mechanism in long short-term memory
(LSTM) language models can capture power law decay. We found that unit
timescales within an LSTM, which are determined by the forget gate bias, should
follow an Inverse Gamma distribution. Experiments then showed that LSTM
language models trained on natural English text learn to approximate this
theoretical distribution. Further, we found that explicitly imposing the
theoretical distribution upon the model during training yielded better language
model perplexity overall, with particular improvements for predicting
low-frequency (rare) words. Moreover, the explicit multi-timescale model
selectively routes information about different types of words through units
with different timescales, potentially improving model interpretability. These
results demonstrate the importance of careful, theoretically-motivated analysis
of memory and timescale in language models.
