  Existing humor datasets and evaluations predominantly focus on English,
lacking resources for culturally nuanced humor in non-English languages like
Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo
Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing
intellectually challenging and culturally specific jokes. We annotate
explanations for each joke and evaluate human explanations against two
state-of-the-art LLMs, GPT-4o and ERNIE Bot, through A/B testing by native
Chinese speakers. Our evaluation shows that Chumor is challenging even for SOTA
LLMs, and the human explanations for Chumor jokes are significantly better than
explanations generated by the LLMs.
