  Text generation with beam search has proven successful in a wide range of
applications. We point out that, though largely overlooked in the literature,
the commonly-used implementation of beam decoding (e.g., Hugging Face
Transformers and fairseq) uses a first come, first served heuristic: it keeps a
set of already completed sequences over time steps and stops when the size of
this set reaches the beam size. Based on this finding, we introduce a patience
factor, a simple modification to this beam decoding implementation, that
generalizes the stopping criterion and provides flexibility to the depth of
search. Empirical results demonstrate that adjusting this patience factor
improves decoding performance of strong pretrained models on news text
summarization and machine translation over diverse language pairs, with a
negligible inference slowdown. Our approach only modifies one line of code and
can be thus readily incorporated in any implementation. Further, we find that
different versions of beam decoding result in large performance differences in
summarization, demonstrating the need for clarity in specifying the beam search
implementation in research work. Our code will be available upon publication.
