  Producing the required amounts of training data for machine learning and NLP
tasks often involves human annotators doing very repetitive and monotonous
work. In this paper, we present and evaluate our novel annotation framework
DALPHI, which facilitates the annotation process by providing the annotator
with suggestions generated by an automated, active-learning based assistance
system. In a study with 66 participants, we demonstrate on the exemplary task
of annotating named entities in text documents that with this assistance system
the annotation processes can be improved with respect to the quality and
quantity of produced annotations, even if the pre-annotations provided by the
assistance system are at a recall level of only 50%.
