  The generation of synthetic images is currently being dominated by Generative
Adversarial Networks (GANs). Despite their outstanding success in generating
realistic looking images, they still suffer from major drawbacks, including an
unstable and highly sensitive training procedure, mode-collapse and
mode-mixture, and dependency on large training sets. In this work we present a
novel non-adversarial generative method - Clustered Optimization of LAtent
space (COLA), which overcomes some of the limitations of GANs, and outperforms
GANs when training data is scarce. In the full data regime, our method is
capable of generating diverse multi-class images with no supervision,
surpassing previous non-adversarial methods in terms of image quality and
diversity. In the small-data regime, where only a small sample of labeled
images is available for training with no access to additional unlabeled data,
our results surpass state-of-the-art GAN models trained on the same amount of
data. Finally, when utilizing our model to augment small datasets, we surpass
the state-of-the-art performance in small-sample classification tasks on
challenging datasets, including CIFAR-10, CIFAR-100, STL-10 and Tiny-ImageNet.
A theoretical analysis supporting the essence of the method is presented.
