  Clinical settings are often characterized by abundant unlabelled data and
limited labelled data. This is typically driven by the high burden placed on
oracles (e.g., physicians) to provide annotations. One way to mitigate this
burden is via active learning (AL) which involves the (a) acquisition and (b)
annotation of informative unlabelled instances. Whereas previous work addresses
either one of these elements independently, we propose an AL framework that
addresses both. For acquisition, we propose Bayesian Active Learning by
Consistency (BALC), a sub-framework which perturbs both instances and network
parameters and quantifies changes in the network output probability
distribution. For annotation, we propose SoQal, a sub-framework that
dynamically determines whether, for each acquired unlabelled instance, to
request a label from an oracle or to pseudo-label it instead. We show that BALC
can outperform start-of-the-art acquisition functions such as BALD, and SoQal
outperforms baseline methods even in the presence of a noisy oracle.
