  We reinterpret multiplicative noise in neural networks as auxiliary random
variables that augment the approximate posterior in a variational setting for
Bayesian neural networks. We show that through this interpretation it is both
efficient and straightforward to improve the approximation by employing
normalizing flows while still allowing for local reparametrizations and a
tractable lower bound. In experiments we show that with this new approximation
we can significantly improve upon classical mean field for Bayesian neural
networks on both predictive accuracy as well as predictive uncertainty.
