  Given a prefix (context), open-ended generation aims to decode texts that are
coherent, which do not abruptly drift from previous topics, and informative,
which do not suffer from undesired repetitions. In this paper, we propose
Look-back, an improved decoding algorithm that leverages the Kullback-Leibler
divergence to track the distribution distance between current and historical
decoding steps. Thus Look-back can automatically predict potential repetitive
phrase and topic drift, and remove tokens that may cause the failure modes,
restricting the next token probability distribution within a plausible distance
to the history. We perform decoding experiments on document continuation and
story generation, and demonstrate that Look-back is able to generate more
fluent and coherent text, outperforming other strong decoding methods
significantly in both automatic and human evaluations.
