  Labeling data correctly is an expensive and challenging task in machine
learning, especially for on-line data streams. Deep learning models especially
require a large number of clean labeled data that is very difficult to acquire
in real-world problems. Choosing useful data samples to label while minimizing
the cost of labeling is crucial to maintain efficiency in the training process.
When confronted with multiple labelers with different expertise and respective
labeling costs, deciding which labeler to choose is nontrivial. In this paper,
we consider a novel weak and strong labeler problem inspired by humans natural
ability for labeling, in the presence of data streams with noisy labels and
constrained by a limited budget. We propose an on-line active learning
algorithm that consists of four steps: filtering, adding diversity, informative
sample selection, and labeler selection. We aim to filter out the suspicious
noisy samples and spend the budget on the diverse informative data using strong
and weak labelers in a cost-effective manner. We derive a decision function
that measures the information gain by combining the informativeness of
individual samples and model confidence. We evaluate our proposed algorithm on
the well-known image classification datasets CIFAR10 and CIFAR100 with up to
60% noise. Experiments show that by intelligently deciding which labeler to
query, our algorithm maintains the same accuracy compared to the case of having
only one of the labelers available while spending less of the budget.
