  Large language models, particularly multilingual ones, are designed, claimed,
and expected to cater to native speakers of varied languages. We hypothesise
that the current practices of fine-tuning and evaluating these models may not
perfectly align with this objective owing to a heavy reliance on translation,
which can introduce translation artefacts and defects. It remains unknown
whether the nature of the instruction data has an impact on the model output;
conversely, it is questionable whether translated test sets can capture such
nuances. Due to the often coupled practices of using translated data in both
stages, such imperfections could have been overlooked. This work investigates
these issues using controlled native or translated data during instruction
tuning and evaluation stages. Experiments on eight base models and eight
different benchmarks show that native or generation benchmarks reveal a notable
difference between native and translated instruction data especially when model
performance is high, whereas other types of test sets cannot. The comparison
between round-trip and single-pass translations reflects the importance of
knowledge from language-native resources. Finally, we demonstrate that
regularization is beneficial to bridging this gap on structured but not
generative tasks.
