  We propose a Topic Compositional Neural Language Model (TCNLM), a novel
method designed to simultaneously capture both the global semantic meaning and
the local word ordering structure in a document. The TCNLM learns the global
semantic coherence of a document via a neural topic model, and the probability
of each learned latent topic is further used to build a Mixture-of-Experts
(MoE) language model, where each expert (corresponding to one topic) is a
recurrent neural network (RNN) that accounts for learning the local structure
of a word sequence. In order to train the MoE model efficiently, a matrix
factorization method is applied, by extending each weight matrix of the RNN to
be an ensemble of topic-dependent weight matrices. The degree to which each
member of the ensemble is used is tied to the document-dependent probability of
the corresponding topics. Experimental results on several corpora show that the
proposed approach outperforms both a pure RNN-based model and other
topic-guided language models. Further, our model yields sensible topics, and
also has the capacity to generate meaningful sentences conditioned on given
topics.
