  Despite the huge progress in myriad generation tasks, pretrained language
models (LMs) such as GPT2 still tend to generate repetitive texts with
maximization-based decoding algorithms for open-ended generation. We attribute
their overestimation of token-level repetition probabilities to the learning
bias: LMs capture simple repetitive patterns faster with the MLE loss. We
propose self-contrastive training to penalize the output of a premature
checkpoint of the same model when it incorrectly predicts repetition, which is
shown to mitigate repetition effectively while maintaining fluency on two
datasets. Furthermore, we find that LMs use longer-range dependencies to
predict repetitive tokens than non-repetitive ones, which may be the cause of
sentence-level repetition loops.
