  A new paradigm for machine translation has recently emerged: fine-tuning
large language models (LLM) on parallel text has been shown to outperform
dedicated translation systems trained in a supervised fashion on much larger
amounts of parallel data (Xu et al., 2024a; Alves et al., 2024). However, it
remains unclear whether this paradigm can enable massively multilingual machine
translation or whether it requires fine-tuning dedicated models for a small
number of language pairs. How does translation fine-tuning impact the MT
capabilities of LLMs for zero-shot languages, zero-shot language pairs, and
translation tasks that do not involve English? To address these questions, we
conduct an extensive empirical evaluation of the translation quality of the
TOWER family of language models (Alves et al., 2024) on 132 translation tasks
from the multi-parallel FLORES-200 data. We find that translation fine-tuning
improves translation quality even for zero-shot languages on average, but that
the impact is uneven depending on the language pairs involved. These results
call for further research to effectively enable massively multilingual
translation with LLMs.
