  A crucial problem in learning disentangled image representations is
controlling the degree of disentanglement during image editing, while
preserving the identity of objects. In this work, we propose a simple yet
effective model with the encoder-decoder architecture to address this
challenge. To encourage disentanglement, we devise a distance covariance based
decorrelation regularization. Further, for the reconstruction step, our model
leverages a soft target representation combined with the latent image code. By
exploiting the real-valued space of the soft target representations, we are
able to synthesize novel images with the designated properties. We also design
a classification based protocol to quantitatively evaluate the disentanglement
strength of our model. Experimental results show that the proposed model
competently disentangles factors of variation, and is able to manipulate face
images to synthesize the desired attributes.
