  Human annotators typically provide annotated data for training machine
learning models, such as neural networks. Yet, human annotations are subject to
noise, impairing generalization performances. Methodological research on
approaches counteracting noisy annotations requires corresponding datasets for
a meaningful empirical evaluation. Consequently, we introduce a novel benchmark
dataset, dopanim, consisting of about 15,750 animal images of 15 classes with
ground truth labels. For approximately 10,500 of these images, 20 humans
provided over 52,000 annotations with an accuracy of circa 67%. Its key
attributes include (1) the challenging task of classifying doppelganger
animals, (2) human-estimated likelihoods as annotations, and (3) annotator
metadata. We benchmark well-known multi-annotator learning approaches using
seven variants of this dataset and outline further evaluation use cases such as
learning beyond hard class labels and active learning. Our dataset and a
comprehensive codebase are publicly available to emulate the data collection
process and to reproduce all empirical results.
