  The swift progress and widespread acceptance of artificial intelligence (AI)
systems highlight a pressing requirement to comprehend both the capabilities
and potential risks associated with AI. Given the linguistic complexity,
cultural richness, and underrepresented status of Arabic in AI research, there
is a pressing need to focus on Large Language Models (LLMs) performance and
safety for Arabic related tasks. Despite some progress in their development,
there is a lack of comprehensive trustworthiness evaluation benchmarks which
presents a major challenge in accurately assessing and improving the safety of
LLMs when prompted in Arabic. In this paper, we introduce AraTrust, the first
comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises
516 human-written multiple-choice questions addressing diverse dimensions
related to truthfulness, ethics, safety, physical health, mental health,
unfairness, illegal activities, privacy, and offensive language. We evaluated a
set of LLMs against our benchmark to assess their trustworthiness. GPT-4 was
the most trustworthy LLM, while open-source models, particularly AceGPT 7B and
Jais 13B, struggled to achieve a score of 60% in our benchmark.
