  There is a rapidly increasing interest in crowdsourcing for data labeling. By
crowdsourcing, a large number of labels can be often quickly gathered at low
cost. However, the labels provided by the crowdsourcing workers are usually not
of high quality. In this paper, we propose a minimax conditional entropy
principle to infer ground truth from noisy crowdsourced labels. Under this
principle, we derive a unique probabilistic labeling model jointly
parameterized by worker ability and item difficulty. We also propose an
objective measurement principle, and show that our method is the only method
which satisfies this objective measurement principle. We validate our method
through a variety of real crowdsourcing datasets with binary, multiclass or
ordinal labels.
