  The Laplace approximation has been one of the workhorses of Bayesian
inference. It often delivers good approximations in practice despite the fact
that it does not strictly take into account where the volume of posterior
density lies. Variational approaches avoid this issue by explicitly minimising
the Kullback-Leibler divergence DKL between a postulated posterior and the true
(unnormalised) logarithmic posterior. However, they rely on a closed form DKL
in order to update the variational parameters. To address this, stochastic
versions of variational inference have been devised that approximate the
intractable DKL with a Monte Carlo average. This approximation allows
calculating gradients with respect to the variational parameters. However,
variational methods often postulate a factorised Gaussian approximating
posterior. In doing so, they sacrifice a-posteriori correlations. In this work,
we propose a method that combines the Laplace approximation with the
variational approach. The advantages are that we maintain: applicability on
non-conjugate models, posterior correlations and a reduced number of free
variational parameters. Numerical experiments demonstrate improvement over the
Laplace approximation and variational inference with factorised Gaussian
posteriors.
