  A major open problem in neural machine translation (NMT) is the translation
of idiomatic expressions, such as "under the weather". The meaning of these
expressions is not composed by the meaning of their constituent words, and NMT
models tend to translate them literally (i.e., word-by-word), which leads to
confusing and nonsensical translations. Research on idioms in NMT is limited
and obstructed by the absence of automatic methods for quantifying these
errors. In this work, first, we propose a novel metric for automatically
measuring the frequency of literal translation errors without human
involvement. Equipped with this metric, we present controlled translation
experiments with models trained in different conditions (with/without the
test-set idioms) and across a wide range of (global and targeted) metrics and
test sets. We explore the role of monolingual pretraining and find that it
yields substantial targeted improvements, even without observing any
translation examples of the test-set idioms. In our analysis, we probe the role
of idiom context. We find that the randomly initialized models are more local
or "myopic" as they are relatively unaffected by variations of the idiom
context, unlike the pretrained ones.
