  Learning to disentangle the hidden factors of variations within a set of
observations is a key task for artificial intelligence. We present a unified
formulation for class and content disentanglement and use it to illustrate the
limitations of current methods. We therefore introduce LORD, a novel method
based on Latent Optimization for Representation Disentanglement. We find that
latent optimization, along with an asymmetric noise regularization, is superior
to amortized inference for achieving disentangled representations. In extensive
experiments, our method is shown to achieve better disentanglement performance
than both adversarial and non-adversarial methods that use the same level of
supervision. We further introduce a clustering-based approach for extending our
method for settings that exhibit in-class variation with promising results on
the task of domain translation.
