  Annotation of training data is the major bottleneck in the creation of text
classification systems. Active learning is a commonly used technique to reduce
the amount of training data one needs to label. A crucial aspect of active
learning is determining when to stop labeling data. Three potential sources for
informing when to stop active learning are an additional labeled set of data,
an unlabeled set of data, and the training data that is labeled during the
process of active learning. To date, no one has compared and contrasted the
advantages and disadvantages of stopping methods based on these three
information sources. We find that stopping methods that use unlabeled data are
more effective than methods that use labeled data.
