  Trans-dimensional random field language models (TRF LMs) have recently been
introduced, where sentences are modeled as a collection of random fields. The
TRF approach has been shown to have the advantages of being computationally
more efficient in inference than LSTM LMs with close performance and being able
to flexibly integrating rich features. In this paper we propose neural TRFs,
beyond of the previous discrete TRFs that only use linear potentials with
discrete features. The idea is to use nonlinear potentials with continuous
features, implemented by neural networks (NNs), in the TRF framework. Neural
TRFs combine the advantages of both NNs and TRFs. The benefits of word
embedding, nonlinear feature learning and larger context modeling are inherited
from the use of NNs. At the same time, the strength of efficient inference by
avoiding expensive softmax is preserved. A number of technical contributions,
including employing deep convolutional neural networks (CNNs) to define the
potentials and incorporating the joint stochastic approximation (JSA) strategy
in the training algorithm, are developed in this work, which enable us to
successfully train neural TRF LMs. Various LMs are evaluated in terms of speech
recognition WERs by rescoring the 1000-best lists of WSJ'92 test data. The
results show that neural TRF LMs not only improve over discrete TRF LMs, but
also perform slightly better than LSTM LMs with only one fifth of parameters
and 16x faster inference efficiency.
