  We study methods for estimating model uncertainty for neural networks (NNs)
in regression. To isolate the effect of model uncertainty, we focus on a
noiseless setting with scarce training data. We introduce five important
desiderata regarding model uncertainty that any method should satisfy. However,
we find that established benchmarks often fail to reliably capture some of
these desiderata, even those that are required by Bayesian theory. To address
this, we introduce a new approach for capturing model uncertainty for NNs,
which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea
of NOMU is to design a network architecture consisting of two connected
sub-NNs, one for model prediction and one for model uncertainty, and to train
it using a carefully-designed loss function. Importantly, our design enforces
that NOMU satisfies our five desiderata. Due to its modular architecture, NOMU
can provide model uncertainty for any given (previously trained) NN if given
access to its training data. We evaluate NOMU in various regressions tasks and
noiseless Bayesian optimization (BO) with costly evaluations. In regression,
NOMU performs at least as well as state-of-the-art methods. In BO, NOMU even
outperforms all considered benchmarks.
