  Warning: this paper contains content that may be offensive or upsetting.
  Most hate speech datasets neglect the cultural diversity within a single
language, resulting in a critical shortcoming in hate speech detection. To
address this, we introduce CREHate, a CRoss-cultural English Hate speech
dataset. To construct CREHate, we follow a two-step procedure: 1) cultural post
collection and 2) cross-cultural annotation. We sample posts from the SBIC
dataset, which predominantly represents North America, and collect posts from
four geographically diverse English-speaking countries (Australia, United
Kingdom, Singapore, and South Africa) using culturally hateful keywords we
retrieve from our survey. Annotations are collected from the four countries
plus the United States to establish representative labels for each country. Our
analysis highlights statistically significant disparities across countries in
hate speech annotations. Only 56.2% of the posts in CREHate achieve consensus
among all countries, with the highest pairwise label difference rate of 26%.
Qualitative analysis shows that label disagreement occurs mostly due to
different interpretations of sarcasm and the personal bias of annotators on
divisive topics. Lastly, we evaluate large language models (LLMs) under a
zero-shot setting and show that current LLMs tend to show higher accuracies on
Anglosphere country labels in CREHate. Our dataset and codes are available at:
https://github.com/nlee0212/CREHate
