  Deep latent-variable models learn representations of high-dimensional data in
an unsupervised manner. A number of recent efforts have focused on learning
representations that disentangle statistically independent axes of variation by
introducing modifications to the standard objective function. These approaches
generally assume a simple diagonal Gaussian prior and as a result are not able
to reliably disentangle discrete factors of variation. We propose a two-level
hierarchical objective to control relative degree of statistical independence
between blocks of variables and individual variables within blocks. We derive
this objective as a generalization of the evidence lower bound, which allows us
to explicitly represent the trade-offs between mutual information between data
and representation, KL divergence between representation and prior, and
coverage of the support of the empirical data distribution. Experiments on a
variety of datasets demonstrate that our objective can not only disentangle
discrete variables, but that doing so also improves disentanglement of other
variables and, importantly, generalization even to unseen combinations of
factors.
