  The posteriors over neural network weights are high dimensional and
multimodal. Each mode typically characterizes a meaningfully different
representation of the data. We develop Cyclical Stochastic Gradient MCMC
(SG-MCMC) to automatically explore such distributions. In particular, we
propose a cyclical stepsize schedule, where larger steps discover new modes,
and smaller steps characterize each mode. We also prove non-asymptotic
convergence of our proposed algorithm. Moreover, we provide extensive
experimental results, including ImageNet, to demonstrate the scalability and
effectiveness of cyclical SG-MCMC in learning complex multimodal distributions,
especially for fully Bayesian inference with modern deep neural networks.
