  Text generation is of great importance to many natural language processing
applications. However, maximization-based decoding methods (e.g. beam search)
of neural language models often lead to degenerate solutions -- the generated
text is unnatural and contains undesirable repetitions. Existing approaches
introduce stochasticity via sampling or modify training objectives to decrease
probabilities of certain tokens (e.g., unlikelihood training). However, they
often lead to solutions that lack coherence. In this work, we show that an
underlying reason for model degeneration is the anisotropic distribution of
token representations. We present a contrastive solution: (i) SimCTG, a
contrastive training objective to calibrate the model's representation space,
and (ii) a decoding method -- contrastive search -- to encourage diversity
while maintaining coherence in the generated text. Extensive experiments and
analyses on three benchmarks from two languages demonstrate that our proposed
approach significantly outperforms current state-of-the-art text generation
methods as evaluated by both human and automatic metrics.
