  To address the global issue of online hate, hate speech detection (HSD)
systems are typically developed on datasets from the United States, thereby
failing to generalize to English dialects from the Majority World. Furthermore,
HSD models are often evaluated on non-representative samples, raising concerns
about overestimating model performance in real-world settings. In this work, we
introduce NaijaHate, the first dataset annotated for HSD which contains a
representative sample of Nigerian tweets. We demonstrate that HSD evaluated on
biased datasets traditionally used in the literature consistently overestimates
real-world performance by at least two-fold. We then propose NaijaXLM-T, a
pretrained model tailored to the Nigerian Twitter context, and establish the
key role played by domain-adaptive pretraining and finetuning in maximizing HSD
performance. Finally, owing to the modest performance of HSD systems in
real-world conditions, we find that content moderators would need to review
about ten thousand Nigerian tweets flagged as hateful daily to moderate 60% of
all hateful content, highlighting the challenges of moderating hate speech at
scale as social media usage continues to grow globally. Taken together, these
results pave the way towards robust HSD systems and a better protection of
social media users from hateful content in low-resource settings.
