  Existing approaches to active learning maximize the system performance by
sampling unlabeled instances for annotation that yield the most efficient
training. However, when active learning is integrated with an end-user
application, this can lead to frustration for participating users, as they
spend time labeling instances that they would not otherwise be interested in
reading. In this paper, we propose a new active learning approach that jointly
optimizes the seemingly counteracting objectives of the active learning system
(training efficiently) and the user (receiving useful instances). We study our
approach in an educational application, which particularly benefits from this
technique as the system needs to rapidly learn to predict the appropriateness
of an exercise to a particular user, while the users should receive only
exercises that match their skills. We evaluate multiple learning strategies and
user types with data from real users and find that our joint approach better
satisfies both objectives when alternative methods lead to many unsuitable
exercises for end users.
