  Representations of data that are invariant to changes in specified factors
are useful for a wide range of problems: removing potential biases in
prediction problems, controlling the effects of covariates, and disentangling
meaningful factors of variation. Unfortunately, learning representations that
exhibit invariance to arbitrary nuisance factors yet remain useful for other
tasks is challenging. Existing approaches cast the trade-off between task
performance and invariance in an adversarial way, using an iterative minimax
optimization. We show that adversarial training is unnecessary and sometimes
counter-productive; we instead cast invariant representation learning as a
single information-theoretic objective that can be directly optimized. We
demonstrate that this approach matches or exceeds performance of
state-of-the-art adversarial approaches for learning fair representations and
for generative modeling with controllable transformations.
