  Information-theoretic approaches to active learning have traditionally
focused on maximising the information gathered about the model parameters, most
commonly by optimising the BALD score. We highlight that this can be suboptimal
from the perspective of predictive performance. For example, BALD lacks a
notion of an input distribution and so is prone to prioritise data of limited
relevance. To address this we propose the expected predictive information gain
(EPIG), an acquisition function that measures information gain in the space of
predictions rather than parameters. We find that using EPIG leads to stronger
predictive performance compared with BALD across a range of datasets and
models, and thus provides an appealing drop-in replacement.
