  Neural Networks (NNs) have been extensively used for a wide spectrum of
real-world regression tasks, where the goal is to predict a numerical outcome
such as revenue, effectiveness, or a quantitative result. In many such tasks,
the point prediction is not enough: the uncertainty (i.e. risk or confidence)
of that prediction must also be estimated. Standard NNs, which are most often
used in such tasks, do not provide uncertainty information. Existing approaches
address this issue by combining Bayesian models with NNs, but these models are
hard to implement, more expensive to train, and usually do not predict as
accurately as standard NNs. In this paper, a new framework (RIO) is developed
that makes it possible to estimate uncertainty in any pretrained standard NN.
The behavior of the NN is captured by modeling its prediction residuals with a
Gaussian Process, whose kernel includes both the NN's input and its output. The
framework is evaluated in twelve real-world datasets, where it is found to (1)
provide reliable estimates of uncertainty, (2) reduce the error of the point
predictions, and (3) scale well to large datasets. Given that RIO can be
applied to any standard NN without modifications to model architecture or
training pipeline, it provides an important ingredient for building real-world
NN applications.
