  We propose a Multiscale Invertible Generative Network (MsIGN) and associated
training algorithm that leverages multiscale structure to solve
high-dimensional Bayesian inference. To address the curse of dimensionality,
MsIGN exploits the low-dimensional nature of the posterior, and generates
samples from coarse to fine scale (low to high dimension) by iteratively
upsampling and refining samples. MsIGN is trained in a multi-stage manner to
minimize the Jeffreys divergence, which avoids mode dropping in
high-dimensional cases. On two high-dimensional Bayesian inverse problems, we
show superior performance of MsIGN over previous approaches in posterior
approximation and multiple mode capture. On the natural image synthesis task,
MsIGN achieves superior performance in bits-per-dimension over baseline models
and yields great interpret-ability of its neurons in intermediate layers.
