  Most current computer vision datasets are composed of disconnected sets, such
as images from different classes. We prove that distributions of this type of
data cannot be represented with a continuous generative network without error.
They can be represented in two ways: With an ensemble of networks or with a
single network with truncated latent space. We show that ensembles are more
desirable than truncated distributions in practice. We construct a regularized
optimization problem that establishes the relationship between a single
continuous GAN, an ensemble of GANs, conditional GANs, and Gaussian Mixture
GANs. This regularization can be computed efficiently, and we show empirically
that our framework has a performance sweet spot which can be found with
hyperparameter tuning. This ensemble framework allows better performance than a
single continuous GAN or cGAN while maintaining fewer total parameters.
