  Human annotated data is the cornerstone of today's artificial intelligence
efforts, yet data labeling processes can be complicated and expensive,
especially when human labelers disagree with each other. The current work
practice is to use majority-voted labels to overrule the disagreement. However,
in the subjective data labeling tasks such as hate speech annotation,
disagreement among individual labelers can be difficult to resolve. In this
paper, we explored why such disagreements occur using a mixed-method approach -
including interviews with experts, concept mapping exercises, and
self-reporting items - to develop a multidimensional scale for distilling the
process of how annotators label a hate speech corpus. We tested this scale with
170 annotators in a hate speech annotation task. Results showed that our scale
can reveal facets of individual differences among annotators (e.g., age,
personality, etc.), and these facets' relationships to an annotator's final
label decision of an instance. We suggest that this work contributes to the
understanding of how humans annotate data. The proposed scale can potentially
improve the value of the currently discarded minority-vote labels.
