  Due to its effectiveness and performance, the Transformer translation model
has attracted wide attention, most recently in terms of probing-based
approaches. Previous work focuses on using or probing source linguistic
features in the encoder. To date, the way word translation evolves in
Transformer layers has not yet been investigated. Naively, one might assume
that encoder layers capture source information while decoder layers translate.
In this work, we show that this is not quite the case: translation already
happens progressively in encoder layers and even in the input embeddings. More
surprisingly, we find that some of the lower decoder layers do not actually do
that much decoding. We show all of this in terms of a probing approach where we
project representations of the layer analyzed to the final trained and frozen
classifier level of the Transformer decoder to measure word translation
accuracy. Our findings motivate and explain a Transformer configuration change:
if translation already happens in the encoder layers, perhaps we can increase
the number of encoder layers, while decreasing the number of decoder layers,
boosting decoding speed, without loss in translation quality? Our experiments
show that this is indeed the case: we can increase speed by up to a factor 2.3
with small gains in translation quality, while an 18-4 deep encoder
configuration boosts translation quality by +1.42 BLEU (En-De) at a speed-up of
1.4.
