  Text generative models trained via Maximum Likelihood Estimation (MLE) suffer
from the notorious exposure bias problem, and Generative Adversarial Networks
(GANs) are shown to have potential to tackle this problem. Existing language
GANs adopt estimators like REINFORCE or continuous relaxations to model word
probabilities. The inherent limitations of such estimators lead current models
to rely on pre-training techniques (MLE pre-training or pre-trained
embeddings). Representation modeling methods which are free from those
limitations, however, are seldomly explored because of their poor performance
in previous attempts. Our analyses reveal that invalid sampling methods and
unhealthy gradients are the main contributors to such unsatisfactory
performance. In this work, we present two techniques to tackle these problems:
dropout sampling and fully normalized LSTM. Based on these two techniques, we
propose InitialGAN whose parameters are randomly initialized in full. Besides,
we introduce a new evaluation metric, Least Coverage Rate, to better evaluate
the quality of generated samples. The experimental results demonstrate that
InitialGAN outperforms both MLE and other compared models. To the best of our
knowledge, it is the first time a language GAN can outperform MLE without using
any pre-training techniques.
