  Scholarship on generative pretraining (GPT) remains acutely Anglocentric,
leaving serious gaps in our understanding of the whole class of autoregressive
models. For example, we have little knowledge about the potential of these
models and their societal impacts in diverse linguistic and cultural settings.
We alleviate this issue for Arabic, a wide collection of languages and
dialectal varieties with more than 400 million population, by introducing
JASMINE. JASMINE is a suite of powerful Arabic autoregressive Transformer
language models ranging in size between 300 million-6.7 billion parameters
pretrained on a large and diverse dataset (~ 235 GB of text). We also carefully
design and release a comprehensive benchmark for both automated and human
evaluation of Arabic autoregressive models, with coverage of potential social
biases, harms, and toxicity. Using our novel benchmark, we evaluate JASMINE
extensively showing powerful performance intrinsically as well as in few-shot
learning on a wide range of NLP tasks. We aim to responsibly release our models
and evaluation benchmark with interested researchers, along with code for
experimenting with them.
