  The text generated by neural language models is not as good as the real text.
This means that their distributions are different. Generative Adversarial Nets
(GAN) are used to alleviate it. However, some researchers argue that GAN
variants do not work at all. When both sample quality (such as Bleu) and sample
diversity (such as self-Bleu) are taken into account, the GAN variants even are
worse than a well-adjusted language model. But, Bleu and self-Bleu can not
precisely measure this distributional discrepancy. In fact, how to measure the
distributional discrepancy between real text and generated text is still an
open problem. In this paper, we theoretically propose two metric functions to
measure the distributional difference between real text and generated text.
Besides that, a method is put forward to estimate them. First, we evaluate
language model with these two functions and find the difference is huge. Then,
we try several methods to use the detected discrepancy signal to improve the
generator. However the difference becomes even bigger than before.
Experimenting on two existing language GANs, the distributional discrepancy
between real text and generated text increases with more adversarial learning
rounds. It demonstrates both of these language GANs fail.
