  The rapid evolution of large language models necessitates effective
benchmarks for evaluating their role knowledge, which is essential for
establishing connections with the real world and providing more immersive
interactions. This paper introduces RoleEval, a bilingual benchmark designed to
assess the memorization, utilization, and reasoning capabilities of role
knowledge. RoleEval comprises RoleEval-Global (including internationally
recognized characters) and RoleEval-Chinese (including characters popular in
China), with 6,000 Chinese-English parallel multiple-choice questions focusing
on 300 influential people and fictional characters drawn from a variety of
domains including celebrities, anime, comics, movies, TV series, games, and
fictions. These questions cover basic knowledge and multi-hop reasoning
abilities, aiming to systematically probe various aspects such as personal
information, relationships, abilities, and experiences of the characters. To
maintain high standards, we perform a hybrid quality check process combining
both automatic and human verification, ensuring that the questions are diverse,
challenging, and discriminative.
  Our extensive evaluations with RoleEval across various open-source and
proprietary large language models, under both the zero- and few-shot settings,
reveal insightful findings. Notably, while GPT-4 outperforms other models on
RoleEval-Global, Chinese large language models excel on RoleEval-Chinese,
highlighting significant knowledge distribution differences. We expect that
RoleEval would highlight the significance of assessing role knowledge for large
language models across various languages and cultural settings.
