  The goal of pool-based active learning is to judiciously select a fixed-sized
subset of unlabeled samples from a pool to query an oracle for their labels, in
order to maximize the accuracy of a supervised learner. However, the unsaid
requirement that the oracle should always assign correct labels is unreasonable
for most situations. We propose an active learning technique for deep neural
networks that is more robust to mislabeling than the previously proposed
techniques. Previous techniques rely on the task network itself to estimate the
novelty of the unlabeled samples, but learning the task (generalization) and
selecting samples (out-of-distribution detection) can be conflicting goals. We
use a separate network to score the unlabeled samples for selection. The
scoring network relies on self-supervision for modeling the distribution of the
labeled samples to reduce the dependency on potentially noisy labels. To
counter the paucity of data, we also deploy another head on the scoring network
for regularization via multi-task learning and use an unusual self-balancing
hybrid scoring function. Furthermore, we divide each query into sub-queries
before labeling to ensure that the query has diverse samples. In addition to
having a higher tolerance to mislabeling of samples by the oracle, the
resultant technique also produces competitive accuracy in the absence of label
noise. The technique also handles the introduction of new classes on-the-fly
well by temporarily increasing the sampling rate of these classes.
