  The state of the art in machine translation (MT) is governed by neural
approaches, which typically provide superior translation accuracy over
statistical approaches. However, on the closely related task of word alignment,
traditional statistical word alignment models often remain the go-to solution.
In this paper, we present an approach to train a Transformer model to produce
both accurate translations and alignments. We extract discrete alignments from
the attention probabilities learnt during regular neural machine translation
model training and leverage them in a multi-task framework to optimize towards
translation and alignment objectives. We demonstrate that our approach produces
competitive results compared to GIZA++ trained IBM alignment models without
sacrificing translation accuracy and outperforms previous attempts on
Transformer model based word alignment. Finally, by incorporating IBM model
alignments into our multi-task training, we report significantly better
alignment accuracies compared to GIZA++ on three publicly available data sets.
