  Large-scale pre-trained language models have achieved great success on
natural language generation tasks. However, it is difficult to control the
pre-trained language models to generate sentences with the desired attribute
such as topic and sentiment, etc. Recently, Bayesian Controllable Language
Models (BCLMs) have been shown to be efficient in controllable language
generation. Rather than fine-tuning the parameters of pre-trained language
models, BCLMs use external discriminators to guide the generation of
pre-trained language models. However, the mismatch between training and
inference of BCLMs limits the performance of the models. To address the
problem, in this work we propose a "Gemini Discriminator" for controllable
language generation which alleviates the mismatch problem with a small
computational cost. We tested our method on two controllable language
generation tasks: sentiment control and topic control. On both tasks, our
method reached achieved new state-of-the-art results in automatic and human
evaluations.
