  As larger and more comprehensive datasets become standard in contemporary
machine learning, it becomes increasingly more difficult to obtain reliable,
trustworthy label information with which to train sophisticated models. To
address this problem, crowdsourcing has emerged as a popular, inexpensive, and
efficient data mining solution for performing distributed label collection.
However, crowdsourced annotations are inherently untrustworthy, as the labels
are provided by anonymous volunteers who may have varying, unreliable
expertise. Worse yet, some participants on commonly used platforms such as
Amazon Mechanical Turk may be adversarial, and provide intentionally incorrect
label information without the end user's knowledge. We discuss three
conventional models of the label generation process, describing their
parameterizations and the model-based approaches used to solve them. We then
propose OpinionRank, a model-free, interpretable, graph-based spectral
algorithm for integrating crowdsourced annotations into reliable labels for
performing supervised or semi-supervised learning. Our experiments show that
OpinionRank performs favorably when compared against more highly parameterized
algorithms. We also show that OpinionRank is scalable to very large datasets
and numbers of label sources, and requires considerably fewer computational
resources than previous approaches.
