  Lexically constrained sentence generation allows the incorporation of prior
knowledge such as lexical constraints into the output. This technique has been
applied to machine translation, and dialog response generation. Previous work
usually used Markov Chain Monte Carlo (MCMC) sampling to generate lexically
constrained sentences, but they randomly determined the position to be edited
and the action to be taken, resulting in many invalid refinements. To overcome
this challenge, we used a classifier to instruct the MCMC-based models where
and how to refine the candidate sentences. First, we developed two methods to
create synthetic data on which the pre-trained model is fine-tuned to obtain a
reliable classifier. Next, we proposed a two-step approach, "Predict and
Revise", for constrained sentence generation. During the predict step, we
leveraged the classifier to compute the learned prior for the candidate
sentence. During the revise step, we resorted to MCMC sampling to revise the
candidate sentence by conducting a sampled action at a sampled position drawn
from the learned prior. We compared our proposed models with many strong
baselines on two tasks, generating sentences with lexical constraints and text
infilling. Experimental results have demonstrated that our proposed model
performs much better than the previous work in terms of sentence fluency and
diversity. Our code and pre-trained models are available at
https://github.com/NLPCode/MCMCXLNet.
