  Pre-training models on vast quantities of unlabeled data has emerged as an
effective approach to improving accuracy on many NLP tasks. On the other hand,
traditional machine translation has a long history of leveraging unlabeled data
through noisy channel modeling. The same idea has recently been shown to
achieve strong improvements for neural machine translation. Unfortunately,
na\"{i}ve noisy channel modeling with modern sequence to sequence models is up
to an order of magnitude slower than alternatives. We address this issue by
introducing efficient approximations to make inference with the noisy channel
approach as fast as strong ensembles while increasing accuracy. We also show
that the noisy channel approach can outperform strong pre-training results by
achieving a new state of the art on WMT Romanian-English translation.
