  As crowdsourcing emerges as an efficient and cost-effective method for
obtaining labels for machine learning datasets, it is important to assess the
quality of crowd-provided data, so as to improve analysis performance and
reduce biases in subsequent machine learning tasks. Given the lack of ground
truth in most cases of crowdsourcing, we refer to data quality as annotators'
consistency and credibility. Unlike the simple scenarios where Kappa
coefficient and intraclass correlation coefficient usually can apply, online
crowdsourcing requires dealing with more complex situations. We introduce a
systematic method for evaluating data quality and detecting spamming threats
via variance decomposition, and we classify spammers into three categories
based on their different behavioral patterns. A spammer index is proposed to
assess entire data consistency and two metrics are developed to measure crowd
worker's credibility by utilizing the Markov chain and generalized random
effects models. Furthermore, we showcase the practicality of our techniques and
their advantages by applying them on a face verification task with both
simulation and real-world data collected from two crowdsourcing platforms.
