  Well-annotated data is a prerequisite for good Natural Language Processing
models. Too often, though, annotation decisions are governed by optimizing time
or annotator agreement. We make a case for nuanced efforts in an
interdisciplinary setting for annotating offensive online speech. Detecting
offensive content is rapidly becoming one of the most important real-world NLP
tasks. However, most datasets use a single binary label, e.g., for hate or
incivility, even though each concept is multi-faceted. This modeling choice
severely limits nuanced insights, but also performance. We show that a more
fine-grained multi-label approach to predicting incivility and hateful or
intolerant content addresses both conceptual and performance issues. We release
a novel dataset of over 40,000 tweets about immigration from the US and UK,
annotated with six labels for different aspects of incivility and intolerance.
Our dataset not only allows for a more nuanced understanding of harmful speech
online, models trained on it also outperform or match performance on benchmark
datasets.
