  Disentangled representation learning finds compact, independent and
easy-to-interpret factors of the data. Learning such has been shown to require
an inductive bias, which we explicitly encode in a generative model of images.
Specifically, we propose a model with two latent spaces: one that represents
spatial transformations of the input data, and another that represents the
transformed data. We find that the latter naturally captures the intrinsic
appearance of the data. To realize the generative model, we propose a
Variationally Inferred Transformational Autoencoder (VITAE) that incorporates a
spatial transformer into a variational autoencoder. We show how to perform
inference in the model efficiently by carefully designing the encoders and
restricting the transformation class to be diffeomorphic. Empirically, our
model separates the visual style from digit type on MNIST, separates shape and
pose in images of human bodies and facial features from facial shape on CelebA.
