  Massively multilingual machine translation (MT) has shown impressive
capabilities, including zero and few-shot translation between low-resource
language pairs. However, these models are often evaluated on high-resource
languages with the assumption that they generalize to low-resource ones. The
difficulty of evaluating MT models on low-resource pairs is often due to lack
of standardized evaluation datasets. In this paper, we present MENYO-20k, the
first multi-domain parallel corpus with a special focus on clean orthography
for Yor\`ub\'a--English with standardized train-test splits for benchmarking.
We provide several neural MT benchmarks and compare them to the performance of
popular pre-trained (massively multilingual) MT models both for the
heterogeneous test set and its subdomains. Since these pre-trained models use
huge amounts of data with uncertain quality, we also analyze the effect of
diacritics, a major characteristic of Yor\`ub\'a, in the training data. We
investigate how and when this training condition affects the final quality and
intelligibility of a translation. Our models outperform massively multilingual
models such as Google ($+8.7$ BLEU) and Facebook M2M ($+9.1$ BLEU) when
translating to Yor\`ub\'a, setting a high quality benchmark for future
research.
