  Neural encoder-decoder models of machine translation have achieved impressive
results, rivalling traditional translation models. However their modelling
formulation is overly simplistic, and omits several key inductive biases built
into traditional models. In this paper we extend the attentional neural
translation model to include structural biases from word based alignment
models, including positional bias, Markov conditioning, fertility and agreement
over translation directions. We show improvements over a baseline attentional
model and standard phrase-based model over several language pairs, evaluating
on difficult languages in a low resource setting.
