  Recent language models, especially those based on recurrent neural networks
(RNNs), make it possible to generate natural language from a learned
probability. Language generation has wide applications including machine
translation, summarization, question answering, conversation systems, etc.
Existing methods typically learn a joint probability of words conditioned on
additional information, which is (either statically or dynamically) fed to
RNN's hidden layer. In many applications, we are likely to impose hard
constraints on the generated texts, i.e., a particular word must appear in the
sentence. Unfortunately, existing approaches could not solve this problem. In
this paper, we propose a novel backward and forward language model. Provided a
specific word, we use RNNs to generate previous words and future words, either
simultaneously or asynchronously, resulting in two model variants. In this way,
the given word could appear at any position in the sentence. Experimental
results show that the generated texts are comparable to sequential LMs in
quality.
