  Uncertainty in machine learning models is a timely and vast field of
research. In supervised learning, uncertainty can already occur in the first
stage of the training process, the annotation phase. This scenario is
particularly evident when some instances cannot be definitively classified. In
other words, there is inevitable ambiguity in the annotation step and hence,
not necessarily a "ground truth" associated with each instance. The main idea
of this work is to drop the assumption of a ground truth label and instead
embed the annotations into a multidimensional space. This embedding is derived
from the empirical distribution of annotations in a Bayesian setup, modeled via
a Dirichlet-Multinomial framework. We estimate the model parameters and
posteriors using a stochastic Expectation Maximization algorithm with Markov
Chain Monte Carlo steps. The methods developed in this paper readily extend to
various situations where multiple annotators independently label instances. To
showcase the generality of the proposed approach, we apply our approach to
three benchmark datasets for image classification and Natural Language
Inference. Besides the embeddings, we can investigate the resulting correlation
matrices, which reflect the semantic similarities of the original classes very
well for all three exemplary datasets.
