  Intelligence relies on an agent's knowledge of what it does not know. This
capability can be assessed based on the quality of joint predictions of labels
across multiple inputs. In principle, ensemble-based approaches produce
effective joint predictions, but the computational costs of training large
ensembles can become prohibitive. We introduce the epinet: an architecture that
can supplement any conventional neural network, including large pretrained
models, and can be trained with modest incremental computation to estimate
uncertainty. With an epinet, conventional neural networks outperform very large
ensembles, consisting of hundreds or more particles, with orders of magnitude
less computation. The epinet does not fit the traditional framework of Bayesian
neural networks. To accommodate development of approaches beyond BNNs, such as
the epinet, we introduce the epistemic neural network (ENN) as an interface for
models that produce joint predictions.
