  We formulate sequence to sequence transduction as a noisy channel decoding
problem and use recurrent neural networks to parameterise the source and
channel models. Unlike direct models which can suffer from explaining-away
effects during training, noisy channel models must produce outputs that explain
their inputs, and their component models can be trained with not only paired
training samples but also unpaired samples from the marginal output
distribution. Using a latent variable to control how much of the conditioning
sequence the channel model needs to read in order to generate a subsequent
symbol, we obtain a tractable and effective beam search decoder. Experimental
results on abstractive sentence summarisation, morphological inflection, and
machine translation show that noisy channel models outperform direct models,
and that they significantly benefit from increased amounts of unpaired output
data that direct models cannot easily use.
