  Traditional neural networks are simple to train but they typically produce
overconfident predictions. In contrast, Bayesian neural networks provide good
uncertainty quantification but optimizing them is time consuming due to the
large parameter space. This paper proposes to combine the advantages of both
approaches by performing Variational Inference in the Final layer Output space
(VIFO), because the output space is much smaller than the parameter space. We
use neural networks to learn the mean and the variance of the probabilistic
output. Like standard, non-Beyesian models, VIFO enjoys simple training and one
can use Rademacher complexity to provide risk bounds for the model. On the
other hand, using the Bayesian formulation we incorporate collapsed variational
inference with VIFO which significantly improves the performance in practice.
Experiments show that VIFO and ensembles of VIFO provide a good tradeoff in
terms of run time and uncertainty quantification, especially for out of
distribution data.
