  Numerous datasets have been proposed to combat the spread of online hate.
Despite these efforts, a majority of these resources are English-centric,
primarily focusing on overt forms of hate. This research gap calls for
developing high-quality corpora in diverse languages that also encapsulate more
subtle hate expressions. This study introduces K-HATERS, a new corpus for hate
speech detection in Korean, comprising approximately 192K news comments with
target-specific offensiveness ratings. This resource is the largest offensive
language corpus in Korean and is the first to offer target-specific ratings on
a three-point Likert scale, enabling the detection of hate expressions in
Korean across varying degrees of offensiveness. We conduct experiments showing
the effectiveness of the proposed corpus, including a comparison with existing
datasets. Additionally, to address potential noise and bias in human
annotations, we explore a novel idea of adopting the Cognitive Reflection Test,
which is widely used in social science for assessing an individual's cognitive
ability, as a proxy of labeling quality. Findings indicate that annotations
from individuals with the lowest test scores tend to yield detection models
that make biased predictions toward specific target groups and are less
accurate. This study contributes to the NLP research on hate speech detection
and resource construction. The code and dataset can be accessed at
https://github.com/ssu-humane/K-HATERS.
