  ChatGPT's emergence heralds a transformative phase in NLP, particularly
demonstrated through its excellent performance on many English benchmarks.
However, the model's efficacy across diverse linguistic contexts remains
largely uncharted territory. This work aims to bridge this knowledge gap, with
a primary focus on assessing ChatGPT's capabilities on Arabic languages and
dialectal varieties. Our comprehensive study conducts a large-scale automated
and human evaluation of ChatGPT, encompassing 44 distinct language
understanding and generation tasks on over 60 different datasets. To our
knowledge, this marks the first extensive performance analysis of ChatGPT's
deployment in Arabic NLP. Our findings indicate that, despite its remarkable
performance in English, ChatGPT is consistently surpassed by smaller models
that have undergone finetuning on Arabic. We further undertake a meticulous
comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal
Arabic (DA), unveiling the relative shortcomings of both models in handling
Arabic dialects compared to MSA. Although we further explore and confirm the
utility of employing GPT-4 as a potential alternative for human evaluation, our
work adds to a growing body of research underscoring the limitations of
ChatGPT.
