  Conventional active learning algorithms assume a single labeler that produces
noiseless label at a given, fixed cost, and aim to achieve the best
generalization performance for given classifier under a budget constraint.
However, in many real settings, different labelers have different labeling
costs and can yield different labeling accuracies. Moreover, a given labeler
may exhibit different labeling accuracies for different instances. This setting
can be referred to as active learning with diverse labelers with varying costs
and accuracies, and it arises in many important real settings. It is therefore
beneficial to understand how to effectively trade-off between labeling accuracy
for different instances, labeling costs, as well as the informativeness of
training instances, so as to achieve the best generalization performance at the
lowest labeling cost. In this paper, we propose a new algorithm for selecting
instances, labelers (and their corresponding costs and labeling accuracies),
that employs generalization bound of learning with label noise to select
informative instances and labelers so as to achieve higher generalization
accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art
performance on five UCI and a real crowdsourcing dataset.
