  We propose a general modeling and inference framework that composes
probabilistic graphical models with deep learning methods and combines their
respective strengths. Our model family augments graphical structure in latent
variables with neural network observation models. For inference, we extend
variational autoencoders to use graphical model approximating distributions
with recognition networks that output conjugate potentials. All components of
these models are learned simultaneously with a single objective, giving a
scalable algorithm that leverages stochastic variational inference, natural
gradients, graphical model message passing, and the reparameterization trick.
We illustrate this framework with several example models and an application to
mouse behavioral phenotyping.
