  Generative models have made significant progress in the tasks of modeling
complex data distributions such as natural images. The introduction of
Generative Adversarial Networks (GANs) and auto-encoders lead to the
possibility of training on big data sets in an unsupervised manner. However,
for many generative models it is not possible to specify what kind of image
should be generated and it is not possible to translate existing images into
new images of similar domains. Furthermore, models that can perform
image-to-image translation often need distinct models for each domain, making
it hard to scale these systems to multiple domain image-to-image translation.
We introduce a model that can do both, controllable image generation and
image-to-image translation between multiple domains. We split our image
representation into two parts encoding unstructured and structured information
respectively. The latter is designed in a disentangled manner, so that
different parts encode different image characteristics. We train an encoder to
encode images into these representations and use a small amount of labeled data
to specify what kind of information should be encoded in the disentangled part.
A generator is trained to generate images from these representations using the
characteristics provided by the disentangled part of the representation.
Through this we can control what kind of images the generator generates,
translate images between different domains, and even learn unknown
data-generating factors while only using one single model.
