  Class imbalance is a prevalent issue in real world machine learning
applications, often leading to poor performance in rare and minority classes.
With an abundance of wild unlabeled data, active learning is perhaps the most
effective technique in solving the problem at its root -- collecting a more
balanced and informative set of labeled examples during annotation. Label noise
is another common issue in data annotation jobs, which is especially
challenging for active learning methods. In this work, we conduct the first
study of active learning under both class imbalance and label noise. We propose
a novel algorithm that robustly identifies the class separation threshold and
annotates the most uncertain examples that are closest from it. Through a novel
reduction to one-dimensional active learning, our algorithm DIRECT is able to
leverage the classic active learning literature to address issues such as batch
labeling and tolerance towards label noise. We present extensive experiments on
imbalanced datasets with and without label noise. Our results demonstrate that
DIRECT can save more than 60% of the annotation budget compared to state-of-art
active learning algorithms and more than 80% of annotation budget compared to
random sampling.
