  In this paper, we compare general-purpose pretrained models, GPT-4-Turbo and
Llama-3-8b-Instruct with special-purpose models fine-tuned on specific tasks,
XLM-Roberta-large, mT5-large, and Llama-3-8b-Instruct. We focus on seven
classification and six generation tasks to evaluate the performance of these
models on Urdu language. Urdu has 70 million native speakers, yet it remains
underrepresented in Natural Language Processing (NLP). Despite the frequent
advancements in Large Language Models (LLMs), their performance in low-resource
languages, including Urdu, still needs to be explored. We also conduct a human
evaluation for the generation tasks and compare the results with the
evaluations performed by GPT-4-Turbo and Llama-3-8b-Instruct. We find that
special-purpose models consistently outperform general-purpose models across
various tasks. We also find that the evaluation done by GPT-4-Turbo for
generation tasks aligns more closely with human evaluation compared to the
evaluation by Llama-3-8b-Instruct. This paper contributes to the NLP community
by providing insights into the effectiveness of general and specific-purpose
LLMs for low-resource languages.
