  Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing
but often suffers from poor zero-shot (ZS) translation qualities. While prior
work has explored the causes of overall low ZS performance, our work introduces
a fresh perspective: the presence of high variations in ZS performance. This
suggests that MNMT does not uniformly exhibit poor ZS capability; instead,
certain translation directions yield reasonable results. Through systematic
experimentation involving 1,560 language directions spanning 40 languages, we
identify three key factors contributing to high variations in ZS NMT
performance: 1) target side translation capability 2) vocabulary overlap 3)
linguistic properties. Our findings highlight that the target side translation
quality is the most influential factor, with vocabulary overlap consistently
impacting ZS performance. Additionally, linguistic properties, such as language
family and writing system, play a role, particularly with smaller models.
Furthermore, we suggest that the off-target issue is a symptom of inadequate ZS
performance, emphasizing that zero-shot translation challenges extend beyond
addressing the off-target problem. We release the data and models serving as a
benchmark to study zero-shot for future research at
https://github.com/Smu-Tan/ZS-NMT-Variations
