  One of the most popular estimation methods in Bayesian neural networks (BNN)
is mean-field variational inference (MFVI). In this work, we show that neural
networks with ReLU activation function induce posteriors, that are hard to fit
with MFVI. We provide a theoretical justification for this phenomenon, study it
empirically, and report the results of a series of experiments to investigate
the effect of activation function on the calibration of BNNs. We find that
using Leaky ReLU activations leads to more Gaussian-like weight posteriors and
achieves a lower expected calibration error (ECE) than its ReLU-based
counterpart.
