  We propose a new, two-step empirical Bayes-type of approach for neural
networks. We show in context of the nonparametric regression model that the
procedure (up to a logarithmic factor) provides optimal recovery of the
underlying functional parameter of interest and provides Bayesian credible sets
with frequentist coverage guarantees. The approach requires fitting the neural
network only once, hence it is substantially faster than Bootstrapping type
approaches. We demonstrate the applicability of our method over synthetic data,
observing good estimation properties and reliable uncertainty quantification.
