  Interactive learning is a process in which a machine learning algorithm is
provided with meaningful, well-chosen examples as opposed to randomly chosen
examples typical in standard supervised learning. In this paper, we propose a
new method for interactive learning from multiple noisy labels where we exploit
the disagreement among annotators to quantify the easiness (or meaningfulness)
of an example. We demonstrate the usefulness of this method in estimating the
parameters of a latent variable classification model, and conduct experimental
analyses on a range of synthetic and benchmark datasets. Furthermore, we
theoretically analyze the performance of perceptron in this interactive
learning framework.
