  Language models (LMs) are statistical models that calculate probabilities
over sequences of words or other discrete symbols. Currently two major
paradigms for language modeling exist: count-based n-gram models, which have
advantages of scalability and test-time speed, and neural LMs, which often
achieve superior modeling performance. We demonstrate how both varieties of
models can be unified in a single modeling framework that defines a set of
probability distributions over the vocabulary of words, and then dynamically
calculates mixture weights over these distributions. This formulation allows us
to create novel hybrid models that combine the desirable features of
count-based and neural LMs, and experiments demonstrate the advantages of these
approaches.
