  Most existing methods for conditional image synthesis are only able to
generate a single plausible image for any given input, or at best a fixed
number of plausible images. In this paper, we focus on the problem of
generating images from semantic segmentation maps and present a simple new
method that can generate an arbitrary number of images with diverse appearance
for the same semantic layout. Unlike most existing approaches which adopt the
GAN framework, our method is based on the recently introduced Implicit Maximum
Likelihood Estimation (IMLE) framework. Compared to the leading approach, our
method is able to generate more diverse images while producing fewer artifacts
despite using the same architecture. The learned latent space also has sensible
structure despite the lack of supervision that encourages such behaviour.
Videos and code are available at
https://people.eecs.berkeley.edu/~ke.li/projects/imle/scene_layouts/.
