  Annotating data for supervised learning can be costly. When the annotation
budget is limited, active learning can be used to select and annotate those
observations that are likely to give the most gain in model performance. We
propose an active learning algorithm that, in addition to selecting which
observation to annotate, selects the precision of the annotation that is
acquired. Assuming that annotations with low precision are cheaper to obtain,
this allows the model to explore a larger part of the input space, with the
same annotation budget. We build our acquisition function on the previously
proposed BALD objective for Gaussian Processes, and empirically demonstrate the
gains of being able to adjust the annotation precision in the active learning
loop.
