  Recent advances in conditional image generation tasks, such as image-to-image
translation and image inpainting, are largely accounted to the success of
conditional GAN models, which are often optimized by the joint use of the GAN
loss with the reconstruction loss. However, we reveal that this training recipe
shared by almost all existing methods causes one critical side effect: lack of
diversity in output samples. In order to accomplish both training stability and
multimodal output generation, we propose novel training schemes with a new set
of losses named moment reconstruction losses that simply replace the
reconstruction loss. We show that our approach is applicable to any conditional
generation tasks by performing thorough experiments on image-to-image
translation, super-resolution and image inpainting using Cityscapes and CelebA
dataset. Quantitative evaluations also confirm that our methods achieve a great
diversity in outputs while retaining or even improving the visual fidelity of
generated samples.
