  A Bayesian treatment can mitigate overconfidence in ReLU nets around the
training data. But far away from them, ReLU Bayesian neural networks (BNNs) can
still underestimate uncertainty and thus be asymptotically overconfident. This
issue arises since the output variance of a BNN with finitely many features is
quadratic in the distance from the data region. Meanwhile, Bayesian linear
models with ReLU features converge, in the infinite-width limit, to a
particular Gaussian process (GP) with a variance that grows cubically so that
no asymptotic overconfidence can occur. While this may seem of mostly
theoretical interest, in this work, we show that it can be used in practice to
the benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via
the GP and show that the resulting model is asymptotically maximally uncertain
far away from the data while the BNNs' predictive power is unaffected near the
data. Although the resulting model approximates a full GP posterior, thanks to
its structure, it can be applied \emph{post-hoc} to any pre-trained ReLU BNN at
a low cost.
