  While MCMC methods have become a main work-horse for Bayesian inference,
scaling them to large distributed datasets is still a challenge. Embarrassingly
parallel MCMC strategies take a divide-and-conquer stance to achieve this by
writing the target posterior as a product of subposteriors, running MCMC for
each of them in parallel and subsequently combining the results. The challenge
then lies in devising efficient aggregation strategies. Current strategies
trade-off between approximation quality, and costs of communication and
computation. In this work, we introduce a novel method that addresses these
issues simultaneously. Our key insight is to introduce a deep invertible
transformation to approximate each of the subposteriors. These approximations
can be made accurate even for complex distributions and serve as intermediate
representations, keeping the total communication cost limited. Moreover, they
enable us to sample from the product of the subposteriors using an efficient
and stable importance sampling scheme. We demonstrate the approach outperforms
available state-of-the-art methods in a range of challenging scenarios,
including high-dimensional and heterogeneous subposteriors.
