  Variational Autoencoders (VAEs) are a popular generative model, but one in
which conditional inference can be challenging. If the decomposition into query
and evidence variables is fixed, conditional VAEs provide an attractive
solution. To support arbitrary queries, one is generally reduced to Markov
Chain Monte Carlo sampling methods that can suffer from long mixing times. In
this paper, we propose an idea we term cross-coding to approximate the
distribution over the latent variables after conditioning on an evidence
assignment to some subset of the variables. This allows generating query
samples without retraining the full VAE. We experimentally evaluate three
variations of cross-coding showing that (i) they can be quickly optimized for
different decompositions of evidence and query and (ii) they quantitatively and
qualitatively outperform Hamiltonian Monte Carlo.
