  At its core, this thesis aims to enhance the practicality of deep learning by
improving the label and training efficiency of deep learning models. To this
end, we investigate data subset selection techniques, specifically active
learning and active sampling, grounded in information-theoretic principles.
Active learning improves label efficiency, while active sampling enhances
training efficiency. Supervised deep learning models often require extensive
training with labeled data. Label acquisition can be expensive and
time-consuming, and training large models is resource-intensive, hindering the
adoption outside academic research and "big tech." Existing methods for data
subset selection in deep learning often rely on heuristics or lack a principled
information-theoretic foundation. In contrast, this thesis examines several
objectives for data subset selection and their applications within deep
learning, striving for a more principled approach inspired by information
theory. We begin by disentangling epistemic and aleatoric uncertainty in single
forward-pass deep neural networks, which provides helpful intuitions and
insights into different forms of uncertainty and their relevance for data
subset selection. We then propose and investigate various approaches for active
learning and data subset selection in (Bayesian) deep learning. Finally, we
relate various existing and proposed approaches to approximations of
information quantities in weight or prediction space. Underpinning this work is
a principled and practical notation for information-theoretic quantities that
includes both random variables and observed outcomes. This thesis demonstrates
the benefits of working from a unified perspective and highlights the potential
impact of our contributions to the practical application of deep learning.
