  Classification algorithms aim to predict an unknown label (e.g., a quality
class) for a new instance (e.g., a product). Therefore, training samples
(instances and labels) are used to deduct classification hypotheses. Often, it
is relatively easy to capture instances but the acquisition of the
corresponding labels remain difficult or expensive. Active learning algorithms
select the most beneficial instances to be labeled to reduce cost. In research,
this labeling procedure is simulated and therefore a ground truth is available.
But during deployment, active learning is a one-shot problem and an evaluation
set is not available. Hence, it is not possible to reliably estimate the
performance of the classification system during learning and it is difficult to
decide when the system fulfills the quality requirements (stopping criteria).
In this article, we formalize the task and review existing strategies to assess
the performance of an actively trained classifier during training. Furthermore,
we identified three major challenges: 1)~to derive a performance distribution,
2)~to preserve representativeness of the labeled subset, and 3) to correct
against sampling bias induced by an intelligent selection strategy. In a
qualitative analysis, we evaluate different existing approaches and show that
none of them reliably estimates active learning performance stating a major
challenge for future research for such systems. All plots and experiments are
provided in a Jupyter notebook that is available for download.
