  In this work we present BOTS-LM, a series of bilingual language models
proficient in both Setswana and English. Leveraging recent advancements in data
availability and efficient fine-tuning, BOTS-LM achieves performance similar to
models significantly larger than itself while maintaining computational
efficiency. Our initial release features an 8 billion parameter generative
large language model, with upcoming 0.5 billion and 1 billion parameter large
language models and a 278 million parameter encoder-only model soon to be
released. We find the 8 billion parameter model significantly outperforms
Llama-3-70B and Aya 23 on English-Setswana translation tasks, approaching the
performance of dedicated machine translation models, while approaching 70B
parameter performance on Setswana reasoning as measured by a machine translated
subset of the MMLU benchmark. To accompany the BOTS-LM series of language
models, we release the largest Setswana web dataset, SetsText, totalling over
267 million tokens. In addition, we release the largest machine translated
Setswana dataset, the first and largest synthetic Setswana dataset, training
and evaluation code, training logs, and MMLU-tsn, a machine translated subset
of MMLU.
