  Despite its successes in various machine learning and data science tasks,
crowdsourcing can be susceptible to attacks from dedicated adversaries. This
work investigates the effects of adversaries on crowdsourced classification,
under the popular Dawid and Skene model. The adversaries are allowed to deviate
arbitrarily from the considered crowdsourcing model, and may potentially
cooperate. To address this scenario, we develop an approach that leverages the
structure of second-order moments of annotator responses, to identify large
numbers of adversaries, and mitigate their impact on the crowdsourcing task.
The potential of the proposed approach is empirically demonstrated on synthetic
and real crowdsourcing datasets.
