  This paper presents the Imputer, a neural sequence model that generates
output sequences iteratively via imputations. The Imputer is an iterative
generative model, requiring only a constant number of generation steps
independent of the number of input or output tokens. The Imputer can be trained
to approximately marginalize over all possible alignments between the input and
output sequences, and all possible generation orders. We present a tractable
dynamic programming training algorithm, which yields a lower bound on the log
marginal likelihood. When applied to end-to-end speech recognition, the Imputer
outperforms prior non-autoregressive models and achieves competitive results to
autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1
WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.
