  We introduce a new framework for training deep generative models for
high-dimensional conditional density estimation. The Bottleneck Conditional
Density Estimator (BCDE) is a variant of the conditional variational
autoencoder (CVAE) that employs layer(s) of stochastic variables as the
bottleneck between the input $x$ and target $y$, where both are
high-dimensional. Crucially, we propose a new hybrid training method that
blends the conditional generative model with a joint generative model. Hybrid
blending is the key to effective training of the BCDE, which avoids overfitting
and provides a novel mechanism for leveraging unlabeled data. We show that our
hybrid training procedure enables models to achieve competitive results in the
MNIST quadrant prediction task in the fully-supervised setting, and sets new
benchmarks in the semi-supervised regime for MNIST, SVHN, and CelebA.
