  Human variation in labeling is often considered noise. Annotation projects
for machine learning (ML) aim at minimizing human label variation, with the
assumption to maximize data quality and in turn optimize and maximize machine
learning metrics. However, this conventional practice assumes that there exists
a ground truth, and neglects that there exists genuine human variation in
labeling due to disagreement, subjectivity in annotation or multiple plausible
answers. In this position paper, we argue that this big open problem of human
label variation persists and critically needs more attention to move our field
forward. This is because human label variation impacts all stages of the ML
pipeline: data, modeling and evaluation. However, few works consider all of
these dimensions jointly; and existing research is fragmented. We reconcile
different previously proposed notions of human label variation, provide a
repository of publicly-available datasets with un-aggregated labels, depict
approaches proposed so far, identify gaps and suggest ways forward. As datasets
are becoming increasingly available, we hope that this synthesized view on the
'problem' will lead to an open discussion on possible strategies to devise
fundamentally new directions.
