  Differently from the traditional statistical MT that decomposes the
translation task into distinct separately learned components, neural machine
translation uses a single neural network to model the entire translation
process. Despite neural machine translation being de-facto standard, it is
still not clear how NMT models acquire different competences over the course of
training, and how this mirrors the different models in traditional SMT. In this
work, we look at the competences related to three core SMT components and find
that during training, NMT first focuses on learning target-side language
modeling, then improves translation quality approaching word-by-word
translation, and finally learns more complicated reordering patterns. We show
that this behavior holds for several models and language pairs. Additionally,
we explain how such an understanding of the training process can be useful in
practice and, as an example, show how it can be used to improve vanilla
non-autoregressive neural machine translation by guiding teacher model
selection.
