  A major challenge in sample-based inference (SBI) for Bayesian neural
networks is the size and structure of the networks' parameter space. Our work
shows that successful SBI is possible by embracing the characteristic
relationship between weight and function space, uncovering a systematic link
between overparameterization and the difficulty of the sampling problem.
Through extensive experiments, we establish practical guidelines for sampling
and convergence diagnosis. As a result, we present a deep ensemble initialized
approach as an effective solution with competitive performance and uncertainty
quantification.
