  Generative Adversarial Networks (GANs) have made great progress in
synthesizing realistic images in recent years. However, they are often trained
on image datasets with either too few samples or too many classes belonging to
different data distributions. Consequently, GANs are prone to underfitting or
overfitting, making the analysis of them difficult and constrained. Therefore,
in order to conduct a thorough study on GANs while obviating unnecessary
interferences introduced by the datasets, we train them on artificial datasets
where there are infinitely many samples and the real data distributions are
simple, high-dimensional and have structured manifolds. Moreover, the
generators are designed such that optimal sets of parameters exist.
Empirically, we find that under various distance measures, the generator fails
to learn such parameters with the GAN training procedure. We also find that
training mixtures of GANs leads to more performance gain compared to increasing
the network depth or width when the model complexity is high enough. Our
experimental results demonstrate that a mixture of generators can discover
different modes or different classes automatically in an unsupervised setting,
which we attribute to the distribution of the generation and discrimination
tasks across multiple generators and discriminators. As an example of the
generalizability of our conclusions to realistic datasets, we train a mixture
of GANs on the CIFAR-10 dataset and our method significantly outperforms the
state-of-the-art in terms of popular metrics, i.e., Inception Score (IS) and
Fr\'echet Inception Distance (FID).
