  Variational dimensionality reduction methods are known for their high
accuracy, generative abilities, and robustness. We introduce a framework to
unify many existing variational methods and design new ones. The framework is
based on an interpretation of the multivariate information bottleneck, in which
an encoder graph, specifying what information to compress, is traded-off
against a decoder graph, specifying a generative model. Using this framework,
we rederive existing dimensionality reduction methods including the deep
variational information bottleneck and variational auto-encoders. The framework
naturally introduces a trade-off parameter extending the deep variational CCA
(DVCCA) family of algorithms to beta-DVCCA. We derive a new method, the deep
variational symmetric informational bottleneck (DVSIB), which simultaneously
compresses two variables to preserve information between their compressed
representations. We implement these algorithms and evaluate their ability to
produce shared low dimensional latent spaces on Noisy MNIST dataset. We show
that algorithms that are better matched to the structure of the data (in our
case, beta-DVCCA and DVSIB) produce better latent spaces as measured by
classification accuracy, dimensionality of the latent variables, and sample
efficiency. We believe that this framework can be used to unify other
multi-view representation learning algorithms and to derive and implement novel
problem-specific loss functions.
