  Learning generative models that span multiple data modalities, such as vision
and language, is often motivated by the desire to learn more useful,
generalisable representations that faithfully capture common underlying factors
between the modalities. In this work, we characterise successful learning of
such models as the fulfillment of four criteria: i) implicit latent
decomposition into shared and private subspaces, ii) coherent joint generation
over all modalities, iii) coherent cross-generation across individual
modalities, and iv) improved model learning for individual modalities through
multi-modal integration. Here, we propose a mixture-of-experts multimodal
variational autoencoder (MMVAE) to learn generative models on different sets of
modalities, including a challenging image-language dataset, and demonstrate its
ability to satisfy all four criteria, both qualitatively and quantitatively.
