  Identifying factors that make certain languages harder to model than others
is essential to reach language equality in future Natural Language Processing
technologies. Free-order case-marking languages, such as Russian, Latin or
Tamil, have proved more challenging than fixed-order languages for the tasks of
syntactic parsing and subject-verb agreement prediction. In this work, we
investigate whether this class of languages is also more difficult to translate
by state-of-the-art Neural Machine Translation models (NMT). Using a variety of
synthetic languages and a newly introduced translation challenge set, we find
that word order flexibility in the source language only leads to a very small
loss of NMT quality, even though the core verb arguments become impossible to
disambiguate in sentences without semantic cues. The latter issue is indeed
solved by the addition of case marking. However, in medium- and low-resource
settings, the overall NMT quality of fixed-order languages remains unmatched.
