  Active learning can be defined as iterations of data labeling, model
training, and data acquisition, until sufficient labels are acquired. A
traditional view of data acquisition is that, through iterations, knowledge
from human labels and models is implicitly distilled to monotonically increase
the accuracy and label consistency. Under this assumption, the most recently
trained model is a good surrogate for the current labeled data, from which data
acquisition is requested based on uncertainty/diversity. Our contribution is
debunking this myth and proposing a new objective for distillation. First, we
found example forgetting, which indicates the loss of knowledge learned across
iterations. Second, for this reason, the last model is no longer the best
teacher -- For mitigating such forgotten knowledge, we select one of its
predecessor models as a teacher, by our proposed notion of "consistency". We
show that this novel distillation is distinctive in the following three
aspects; First, consistency ensures to avoid forgetting labels. Second,
consistency improves both uncertainty/diversity of labeled data. Lastly,
consistency redeems defective labels produced by human annotators.
