  We introduce a frustratingly simple, super efficient and surprisingly
effective decoding method, which we call Frustratingly Simple Decoding (FSD),
for neural text generation. The idea behind FSD is straightforward: we build an
anti-LM based on previously generated text and use this anti-LM to penalize
future generation of what has been generated. The anti-LM can be implemented as
simple as an n-gram language model or a vectorized variant. In this way, FSD
introduces no extra model parameters and negligible computational overhead (FSD
can be as fast as greedy search). Despite the simplicity, FSD is surprisingly
effective; Experiments show that FSD can outperform the canonical methods to
date (i.e., nucleus sampling) as well as several strong baselines that were
proposed recently.
