  Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has become
increasingly popular for simulating posterior samples in large-scale Bayesian
modeling. However, existing SG-MCMC schemes are not tailored to any specific
probabilistic model, even a simple modification of the underlying dynamical
system requires significant physical intuition. This paper presents the first
meta-learning algorithm that allows automated design for the underlying
continuous dynamics of an SG-MCMC sampler. The learned sampler generalizes
Hamiltonian dynamics with state-dependent drift and diffusion, enabling fast
traversal and efficient exploration of neural network energy landscapes.
Experiments validate the proposed approach on both Bayesian fully connected
neural network and Bayesian recurrent neural network tasks, showing that the
learned sampler out-performs generic, hand-designed SG-MCMC algorithms, and
generalizes to different datasets and larger architectures.
