  From the intuitive notion of disentanglement, the image variations
corresponding to different factors should be distinct from each other, and the
disentangled representation should reflect those variations with separate
dimensions. To discover the factors and learn disentangled representation,
previous methods typically leverage an extra regularization term when learning
to generate realistic images. However, the term usually results in a trade-off
between disentanglement and generation quality. For the generative models
pretrained without any disentanglement term, the generated images show
semantically meaningful variations when traversing along different directions
in the latent space. Based on this observation, we argue that it is possible to
mitigate the trade-off by $(i)$ leveraging the pretrained generative models
with high generation quality, $(ii)$ focusing on discovering the traversal
directions as factors for disentangled representation learning. To achieve
this, we propose Disentaglement via Contrast (DisCo) as a framework to model
the variations based on the target disentangled representations, and contrast
the variations to jointly discover disentangled directions and learn
disentangled representations. DisCo achieves the state-of-the-art disentangled
representation learning and distinct direction discovering, given pretrained
non-disentangled generative models including GAN, VAE, and Flow. Source code is
at https://github.com/xrenaa/DisCo.
