  Autoencoders exhibit impressive abilities to embed the data manifold into a
low-dimensional latent space, making them a staple of representation learning
methods. However, without explicit supervision, which is often unavailable, the
representation is usually uninterpretable, making analysis and principled
progress challenging. We propose a framework, called latent responses, which
exploits the locally contractive behavior exhibited by variational autoencoders
to explore the learned manifold. More specifically, we develop tools to probe
the representation using interventions in the latent space to quantify the
relationships between latent variables. We extend the notion of disentanglement
to take the learned generative process into account and consequently avoid the
limitations of existing metrics that may rely on spurious correlations. Our
analyses underscore the importance of studying the causal structure of the
representation to improve performance on downstream tasks such as generation,
interpolation, and inference of the factors of variation.
