  We propose a new class of learning algorithms that combines variational
approximation and Markov chain Monte Carlo (MCMC) simulation. Naive algorithms
that use the variational approximation as proposal distribution can perform
poorly because this approximation tends to underestimate the true variance and
other features of the data. We solve this problem by introducing more
sophisticated MCMC algorithms. One of these algorithms is a mixture of two MCMC
kernels: a random walk Metropolis kernel and a blockMetropolis-Hastings (MH)
kernel with a variational approximation as proposaldistribution. The MH kernel
allows one to locate regions of high probability efficiently. The Metropolis
kernel allows us to explore the vicinity of these regions. This algorithm
outperforms variationalapproximations because it yields slightly better
estimates of the mean and considerably better estimates of higher moments, such
as covariances. It also outperforms standard MCMC algorithms because it locates
theregions of high probability quickly, thus speeding up convergence. We
demonstrate this algorithm on the problem of Bayesian parameter estimation for
logistic (sigmoid) belief networks.
