  Parameter Efficient Finetuning (PEFT) has emerged as a viable solution for
improving the performance of Large Language Models (LLMs) without requiring
massive resources and compute. Prior work on multilingual evaluation has shown
that there is a large gap between the performance of LLMs on English and other
languages. Further, there is also a large gap between the performance of
smaller open-source models and larger LLMs. Finetuning can be an effective way
to bridge this gap and make language models more equitable. In this work, we
finetune the LLama-2-7B and Mistral-7B models on two synthetic multilingual
instruction tuning datasets to determine its effect on model performance on six
downstream tasks covering forty languages in all. Additionally, we experiment
with various parameters, such as rank for low-rank adaptation and values of
quantisation to determine their effects on downstream performance and find that
higher rank and higher quantisation values benefit low-resource languages. We
find that PEFT of smaller open-source models sometimes bridges the gap between
the performance of these models and the larger ones, however, English
performance can take a hit. We also find that finetuning sometimes improves
performance on low-resource languages, while degrading performance on
high-resource languages.
