  We focus on the online-based active learning (OAL) setting where an agent
operates over a stream of observations and trades-off between the costly
acquisition of information (labelled observations) and the cost of prediction
errors. We propose a novel foundation for OAL tasks based on partial
monitoring, a theoretical framework specialized in online learning from
partially informative actions. We show that previously studied binary and
multi-class OAL tasks are instances of partial monitoring. We expand the
real-world potential of OAL by introducing a new class of cost-sensitive OAL
tasks. We propose NeuralCBP, the first PM strategy that accounts for predictive
uncertainty with deep neural networks. Our extensive empirical evaluation on
open source datasets shows that NeuralCBP has favorable performance against
state-of-the-art baselines on multiple binary, multi-class and cost-sensitive
OAL tasks.
