  A new whole-sentence language model - neural trans-dimensional random field
language model (neural TRF LM), where sentences are modeled as a collection of
random fields, and the potential function is defined by a neural network, has
been introduced and successfully trained by noise-contrastive estimation (NCE).
In this paper, we extend NCE and propose dynamic noise-contrastive estimation
(DNCE) to solve the two problems observed in NCE training. First, a dynamic
noise distribution is introduced and trained simultaneously to converge to the
data distribution. This helps to significantly cut down the noise sample number
used in NCE and reduce the training cost. Second, DNCE discriminates between
sentences generated from the noise distribution and sentences generated from
the interpolation of the data distribution and the noise distribution. This
alleviates the overfitting problem caused by the sparseness of the training
set. With DNCE, we can successfully and efficiently train neural TRF LMs on
large corpus (about 0.8 billion words) with large vocabulary (about 568 K
words). Neural TRF LMs perform as good as LSTM LMs with less parameters and
being 5x~114x faster in rescoring sentences. Interpolating neural TRF LMs with
LSTM LMs and n-gram LMs can further reduce the error rates.
