  Pre-trained models have achieved remarkable success in natural language
processing (NLP). However, existing pre-training methods underutilize the
benefits of language understanding for generation. Inspired by the idea of
Generative Adversarial Networks (GANs), we propose a GAN-style model for
encoder-decoder pre-training by introducing an auxiliary discriminator,
unifying the ability of language understanding and generation in a single
model. Our model, named as GanLM, is trained with two pre-training objectives:
replaced token detection and replaced token denoising. Specifically, given
masked source sentences, the generator outputs the target distribution and the
discriminator predicts whether the target sampled tokens from distribution are
incorrect. The target sentence is replaced with misclassified tokens to
construct noisy previous context, which is used to generate the gold sentence.
In general, both tasks improve the ability of language understanding and
generation by selectively using the denoising data. Extensive experiments in
language generation benchmarks show that GanLM with the powerful language
understanding capability outperforms various strong pre-trained language models
(PLMs) and achieves state-of-the-art performance.
