  Standard decoding approaches for conditional text generation tasks typically
search for an output hypothesis with high model probability, but this may not
yield the best hypothesis according to human judgments of quality. Reranking to
optimize for "downstream" metrics can better optimize for quality, but many
metrics of interest are computed with pre-trained language models, which are
slow to apply to large numbers of hypotheses. We explore an approach for
reranking hypotheses by using Transformers to efficiently encode lattices of
generated outputs, a method we call EEL. With a single Transformer pass over
the entire lattice, we can approximately compute a contextualized
representation of each token as if it were only part of a single hypothesis in
isolation. We combine this approach with a new class of token-factored
rerankers (TFRs) that allow for efficient extraction of high reranker-scoring
hypotheses from the lattice. Empirically, our approach incurs minimal
degradation error compared to the exponentially slower approach of encoding
each hypothesis individually. When applying EEL with TFRs across three text
generation tasks, our results show both substantial speedup compared to naive
reranking and often better performance on downstream metrics than comparable
approaches.
