  Many computationally-efficient methods for Bayesian deep learning rely on
continuous optimization algorithms, but the implementation of these methods
requires significant changes to existing code-bases. In this paper, we propose
Vprop, a method for Gaussian variational inference that can be implemented with
two minor changes to the off-the-shelf RMSprop optimizer. Vprop also reduces
the memory requirements of Black-Box Variational Inference by half. We derive
Vprop using the conjugate-computation variational inference method, and
establish its connections to Newton's method, natural-gradient methods, and
extended Kalman filters. Overall, this paper presents Vprop as a principled,
computationally-efficient, and easy-to-implement method for Bayesian deep
learning.
