  Labeling real-world datasets is time consuming but indispensable for
supervised machine learning models. A common solution is to distribute the
labeling task across a large number of non-expert workers via crowd-sourcing.
Due to the varying background and experience of crowd workers, the obtained
labels are highly prone to errors and even detrimental to the learning models.
In this paper, we advocate using hybrid intelligence, i.e., combining deep
models and human experts, to design an end-to-end learning framework from noisy
crowd-sourced data, especially in an on-line scenario. We first summarize the
state-of-the-art solutions that address the challenges of noisy labels from
non-expert crowd and learn from multiple annotators. We show how label
aggregation can benefit from estimating the annotators' confusion matrices to
improve the learning process. Moreover, with the help of an expert labeler as
well as classifiers, we cleanse aggregated labels of highly informative samples
to enhance the final classification accuracy. We demonstrate the effectiveness
of our strategies on several image datasets, i.e. UCI and CIFAR-10, using SVM
and deep neural networks. Our evaluation shows that our on-line label
aggregation with confusion matrix estimation reduces the error rate of labels
by over 30%. Furthermore, relabeling only 10% of the data using the expert's
results in over 90% classification accuracy with SVM.
