  High-quality data is crucial for the success of machine learning, but
labeling large datasets is often a time-consuming and costly process. While
semi-supervised learning can help mitigate the need for labeled data, label
quality remains an open issue due to ambiguity and disagreement among
annotators. Thus, we use proposal-guided annotations as one option which leads
to more consistency between annotators. However, proposing a label increases
the probability of the annotators deciding in favor of this specific label.
This introduces a bias which we can simulate and remove. We propose a new
method CleverLabel for Cost-effective LabEling using Validated proposal-guidEd
annotations and Repaired LABELs. CleverLabel can reduce labeling costs by up to
30.0%, while achieving a relative improvement in Kullback-Leibler divergence of
up to 29.8% compared to the previous state-of-the-art on a multi-domain
real-world image classification benchmark. CleverLabel offers a novel solution
to the challenge of efficiently labeling large datasets while also improving
the label quality.
