  When annotators label data, a key metric for quality assurance is
inter-annotator agreement (IAA): the extent to which annotators agree on their
labels. Though many IAA measures exist for simple categorical and ordinal
labeling tasks, relatively little work has considered more complex labeling
tasks, such as structured, multi-object, and free-text annotations.
Krippendorff's alpha, best known for use with simpler labeling tasks, does have
a distance-based formulation with broader applicability, but little work has
studied its efficacy and consistency across complex annotation tasks.
  We investigate the design and evaluation of IAA measures for complex
annotation tasks, with evaluation spanning seven diverse tasks: image bounding
boxes, image keypoints, text sequence tagging, ranked lists, free text
translations, numeric vectors, and syntax trees. We identify the difficulty of
interpretability and the complexity of choosing a distance function as key
obstacles in applying Krippendorff's alpha generally across these tasks. We
propose two novel, more interpretable measures, showing they yield more
consistent IAA measures across tasks and annotation distance functions.
