  Learning controllable and generalizable representation of multivariate data
with desired structural properties remains a fundamental problem in machine
learning. In this paper, we present a novel framework for learning generative
models with various underlying structures in the latent space. We represent the
inductive bias in the form of mask variables to model the dependency structure
in the graphical model and extend the theory of multivariate information
bottleneck to enforce it. Our model provides a principled approach to learn a
set of semantically meaningful latent factors that reflect various types of
desired structures like capturing correlation or encoding invariance, while
also offering the flexibility to automatically estimate the dependency
structure from data. We show that our framework unifies many existing
generative models and can be applied to a variety of tasks including
multi-modal data modeling, algorithmic fairness, and invariant risk
minimization.
