  Current approximate posteriors in Bayesian neural networks (BNNs) exhibit a
crucial limitation: they fail to maintain invariance under reparameterization,
i.e. BNNs assign different posterior densities to different parametrizations of
identical functions. This creates a fundamental flaw in the application of
Bayesian principles as it breaks the correspondence between uncertainty over
the parameters with uncertainty over the parametrized function. In this paper,
we investigate this issue in the context of the increasingly popular linearized
Laplace approximation. Specifically, it has been observed that linearized
predictives alleviate the common underfitting problems of the Laplace
approximation. We develop a new geometric view of reparametrizations from which
we explain the success of linearization. Moreover, we demonstrate that these
reparameterization invariance properties can be extended to the original neural
network predictive using a Riemannian diffusion process giving a
straightforward algorithm for approximate posterior sampling, which empirically
improves posterior fit.
