  Active Learning is a very common yet powerful framework for iteratively and
adaptively sampling subsets of the unlabeled sets with a human in the loop with
the goal of achieving labeling efficiency. Most real world datasets have
imbalance either in classes and slices, and correspondingly, parts of the
dataset are rare. As a result, there has been a lot of work in designing active
learning approaches for mining these rare data instances. Most approaches
assume access to a seed set of instances which contain these rare data
instances. However, in the event of more extreme rareness, it is reasonable to
assume that these rare data instances (either classes or slices) may not even
be present in the seed labeled set, and a critical need for the active learning
paradigm is to efficiently discover these rare data instances. In this work, we
provide an active data discovery framework which can mine unknown data slices
and classes efficiently using the submodular conditional gain and submodular
conditional mutual information functions. We provide a general algorithmic
framework which works in a number of scenarios including image classification
and object detection and works with both rare classes and rare slices present
in the unlabeled set. We show significant accuracy and labeling efficiency
gains with our approach compared to existing state-of-the-art active learning
approaches for actively discovering these rare classes and slices.
