  Many machine learning systems today are trained on large amounts of
human-annotated data. Data annotation tasks that require a high level of
competency make data acquisition expensive, while the resulting labels are
often subjective, inconsistent, and may contain a variety of human biases. To
improve the data quality, practitioners often need to collect multiple
annotations per example and aggregate them before training models. Such a
multi-stage approach results in redundant annotations and may often produce
imperfect "ground truth" that may limit the potential of training accurate
machine learning models. We propose a new end-to-end framework that enables us
to: (i) merge the aggregation step with model training, thus allowing deep
learning systems to learn to predict ground truth estimates directly from the
available data, and (ii) model difficulties of examples and learn
representations of the annotators that allow us to estimate and take into
account their competencies. Our approach is general and has many applications,
including training more accurate models on crowdsourced data, ensemble
learning, as well as classifier accuracy estimation from unlabeled data. We
conduct an extensive experimental evaluation of our method on 5 crowdsourcing
datasets of varied difficulty and show accuracy gains of up to 25% over the
current state-of-the-art approaches for aggregating annotations, as well as
significant reductions in the required annotation redundancy.
