  NLP datasets annotated with human judgments are rife with disagreements
between the judges. This is especially true for tasks depending on subjective
judgments such as sentiment analysis or offensive language detection.
Particularly in these latter cases, the NLP community has come to realize that
the approach of 'reconciling' these different subjective interpretations is
inappropriate. Many NLP researchers have therefore concluded that rather than
eliminating disagreements from annotated corpora, we should preserve
them-indeed, some argue that corpora should aim to preserve all annotator
judgments. But this approach to corpus creation for NLP has not yet been widely
accepted. The objective of the LeWiDi series of shared tasks is to promote this
approach to developing NLP models by providing a unified framework for training
and evaluating with such datasets. We report on the second LeWiDi shared task,
which differs from the first edition in three crucial respects: (i) it focuses
entirely on NLP, instead of both NLP and computer vision tasks in its first
edition; (ii) it focuses on subjective tasks, instead of covering different
types of disagreements-as training with aggregated labels for subjective NLP
tasks is a particularly obvious misrepresentation of the data; and (iii) for
the evaluation, we concentrate on soft approaches to evaluation. This second
edition of LeWiDi attracted a wide array of participants resulting in 13 shared
task submission papers.
