  In recent years, Generative Adversarial Networks have achieved impressive
results in photorealistic image synthesis. This progress nurtures hopes that
one day the classical rendering pipeline can be replaced by efficient models
that are learned directly from images. However, current image synthesis models
operate in the 2D domain where disentangling 3D properties such as camera
viewpoint or object pose is challenging. Furthermore, they lack an
interpretable and controllable representation. Our key hypothesis is that the
image generation process should be modeled in 3D space as the physical world
surrounding us is intrinsically three-dimensional. We define the new task of 3D
controllable image synthesis and propose an approach for solving it by
reasoning both in 3D space and in the 2D image domain. We demonstrate that our
model is able to disentangle latent 3D factors of simple multi-object scenes in
an unsupervised fashion from raw images. Compared to pure 2D baselines, it
allows for synthesizing scenes that are consistent wrt. changes in viewpoint or
object pose. We further evaluate various 3D representations in terms of their
usefulness for this challenging task.
