  Training LLMs for low-resource languages usually utilizes data augmentation
from English using machine translation (MT). This, however, brings a number of
challenges to LLM training: there are large costs attached to translating and
curating huge amounts of content with high-end machine translation solutions;
the translated content carries over cultural biases; and if the translation is
not faithful and accurate, data quality degrades causing issues in the trained
model. In this work, we investigate the role of translation and synthetic data
in training language models. We translate TinyStories, a dataset of 2.2M short
stories for 3-4 year old children, from English to Arabic using the open
NLLB-3B MT model. We train a number of story generation models of size 1M-33M
parameters using this data. We identify a number of quality and task-specific
issues in the resulting models. To rectify these issues, we further pre-train
the models with a small dataset of synthesized high-quality Arabic stories
generated by a capable LLM, representing 1% of the original training data. We
show, using GPT-4 as a judge and Dictionary Learning Analysis from mechanistic
interpretability, that the suggested approach is a practical means to resolve
some of the machine translation pitfalls. We illustrate the improvements
through case studies of linguistic and cultural bias issues.
