  Earlier approaches indirectly studied the information captured by the hidden
states of recurrent and non-recurrent neural machine translation models by
feeding them into different classifiers. In this paper, we look at the encoder
hidden states of both transformer and recurrent machine translation models from
the nearest neighbors perspective. We investigate to what extent the nearest
neighbors share information with the underlying word embeddings as well as
related WordNet entries. Additionally, we study the underlying syntactic
structure of the nearest neighbors to shed light on the role of syntactic
similarities in bringing the neighbors together. We compare transformer and
recurrent models in a more intrinsic way in terms of capturing lexical
semantics and syntactic structures, in contrast to extrinsic approaches used by
previous works. In agreement with the extrinsic evaluations in the earlier
works, our experimental results show that transformers are superior in
capturing lexical semantics, but not necessarily better in capturing the
underlying syntax. Additionally, we show that the backward recurrent layer in a
recurrent model learns more about the semantics of words, whereas the forward
recurrent layer encodes more context.
