  The purpose of unconditional text generation is to train a model with real
sentences, then generate novel sentences of the same quality and diversity as
the training data. However, when different metrics are used for comparing the
methods of unconditional text generation, contradictory conclusions are drawn.
The difficulty is that both the diversity and quality of the sample should be
considered simultaneously when the models are evaluated. To solve this problem,
a novel metric of distributional discrepancy (DD) is designed to evaluate
generators based on the discrepancy between the generated and real training
sentences. However, it cannot compute the DD directly because the distribution
of real sentences is unavailable. Thus, we propose a method for estimating the
DD by training a neural-network-based text classifier. For comparison, three
existing metrics, bi-lingual evaluation understudy (BLEU) versus self-BLEU,
language model score versus reverse language model score, and Fr\'{e}chet
embedding distance, along with the proposed DD, are used to evaluate two
popular generative models of long short-term memory and generative pretrained
transformer 2 on both syntactic and real data. Experimental results show that
DD is significantly better than the three existing metrics for ranking these
generative models.
