  We consider active learning for binary classification in the agnostic
pool-based setting. The vast majority of works in active learning in the
agnostic setting are inspired by the CAL algorithm where each query is
uniformly sampled from the disagreement region of the current version space.
The sample complexity of such algorithms is described by a quantity known as
the disagreement coefficient which captures both the geometry of the hypothesis
space as well as the underlying probability space. To date, the disagreement
coefficient has been justified by minimax lower bounds only, leaving the door
open for superior instance dependent sample complexities. In this work we
propose an algorithm that, in contrast to uniform sampling over the
disagreement region, solves an experimental design problem to determine a
distribution over examples from which to request labels. We show that the new
approach achieves sample complexity bounds that are never worse than the best
disagreement coefficient-based bounds, but in specific cases can be
dramatically smaller. From a practical perspective, the proposed algorithm
requires no hyperparameters to tune (e.g., to control the aggressiveness of
sampling), and is computationally efficient by means of assuming access to an
empirical risk minimization oracle (without any constraints). Empirically, we
demonstrate that our algorithm is superior to state of the art agnostic active
learning algorithms on image classification datasets.
