  We combine conditional variational autoencoders (VAE) with adversarial
censoring in order to learn invariant representations that are disentangled
from nuisance/sensitive variations. In this method, an adversarial network
attempts to recover the nuisance variable from the representation, which the
VAE is trained to prevent. Conditioning the decoder on the nuisance variable
enables clean separation of the representation, since they are recombined for
model learning and data reconstruction. We show this natural approach is
theoretically well-founded with information-theoretic arguments. Experiments
demonstrate that this method achieves invariance while preserving model
learning performance, and results in visually improved performance for style
transfer and generative sampling tasks.
