  Generative models, especially Generative Adversarial Networks (GANs), have
received significant attention recently. However, it has been observed that in
terms of some attributes, e.g. the number of simple geometric primitives in an
image, GANs are not able to learn the target distribution in practice.
Motivated by this observation, we discover two specific problems of GANs
leading to anomalous generalization behaviour, which we refer to as the sample
insufficiency and the pixel-wise combination. For the first problem of sample
insufficiency, we show theoretically and empirically that the batchsize of the
training samples in practice may be insufficient for the discriminator to learn
an accurate discrimination function. It could result in unstable training
dynamics for the generator, leading to anomalous generalization. For the second
problem of pixel-wise combination, we find that besides recognizing the
positive training samples as real, under certain circumstances, the
discriminator could be fooled to recognize the pixel-wise combinations (e.g.
pixel-wise average) of the positive training samples as real. However, those
combinations could be visually different from the real samples in the target
distribution. With the fooled discriminator as reference, the generator would
obtain biased supervision further, leading to the anomalous generalization
behaviour. Additionally, in this paper, we propose methods to mitigate the
anomalous generalization of GANs. Extensive experiments on benchmark show our
proposed methods improve the FID score up to 30\% on natural image dataset.
