  Zero-shot translation aims to translate between language pairs not seen
during training in Multilingual Machine Translation (MMT) and is largely
considered an open problem. A common, albeit resource-consuming, solution is to
add as many related translation directions as possible to the training corpus.
In this paper, we show that for an English-centric model, surprisingly large
zero-shot improvements can be achieved by simply fine-tuning with a very small
amount of multi-parallel data. For example, on the EC30 dataset, we obtain up
to +21.7 ChrF non-English overall improvements (870 directions) by using only
100 multi-parallel samples while preserving English-centric translation
quality. When investigating the size effect of fine-tuning data and its
transfer capabilities, we found that already a small, randomly sampled set of
fine-tuning directions is sufficient to achieve comparable improvements. The
resulting non-English performance is close to the complete translation upper
bound. Even in a minimal setting -- fine-tuning with only one single sample --
the well-known off-target issue is almost completely resolved, explaining parts
-- but not all -- of the observed improvements in translation quality.
