  Annotator disagreement is ubiquitous in natural language processing (NLP)
tasks. There are multiple reasons for such disagreements, including the
subjectivity of the task, difficult cases, unclear guidelines, and so on.
Rather than simply aggregating labels to obtain data annotations, we instead
try to directly model the diverse perspectives of the annotators, and
explicitly account for annotators' idiosyncrasies in the modeling process by
creating representations for each annotator (annotator embeddings) and also
their annotations (annotation embeddings). In addition, we propose TID-8, The
Inherent Disagreement - 8 dataset, a benchmark that consists of eight existing
language understanding datasets that have inherent annotator disagreement. We
test our approach on TID-8 and show that our approach helps models learn
significantly better from disagreements on six different datasets in TID-8
while increasing model size by fewer than 1% parameters. By capturing the
unique tendencies and subjectivity of individual annotators through embeddings,
our representations prime AI models to be inclusive of diverse viewpoints.
