  Instruction tuning has shown great promise in improving the performance of
large language models. However, research on multilingual instruction tuning has
been limited due to the scarcity of high-quality instruction-response datasets
across different languages. To bridge this gap, we present Bactrian-X, a
comprehensive multilingual parallel dataset of 3.4 million instruction-response
pairs across 52 languages. Leveraging this dataset, we train a set of adapters
using low-rank adaptation (LoRA), which are lightweight components that
seamlessly integrate with large language models. These adapters have a
substantially lower parameter count than the base model, making them easily
replaceable and usable as plug-ins for different languages or language groups.
Extensive experiments in various multilingual evaluation settings demonstrate
that models derived from LoRA-based training over Bactrian-X outperform both
the vanilla models and existing instruction-tuned models. The code and models
are publicly available at https://github.com/mbzuai-nlp/bactrian-x
