  Recently, generative adversarial networks (GANs) have shown promising
performance in generating realistic images. However, they often struggle in
learning complex underlying modalities in a given dataset, resulting in
poor-quality generated images. To mitigate this problem, we present a novel
approach called mixture of experts GAN (MEGAN), an ensemble approach of
multiple generator networks. Each generator network in MEGAN specializes in
generating images with a particular subset of modalities, e.g., an image class.
Instead of incorporating a separate step of handcrafted clustering of multiple
modalities, our proposed model is trained through an end-to-end learning of
multiple generators via gating networks, which is responsible for choosing the
appropriate generator network for a given condition. We adopt the categorical
reparameterization trick for a categorical decision to be made in selecting a
generator while maintaining the flow of the gradients. We demonstrate that
individual generators learn different and salient subparts of the data and
achieve a multiscale structural similarity (MS-SSIM) score of 0.2470 for CelebA
and a competitive unsupervised inception score of 8.33 in CIFAR-10.
