  Generative Adversarial Networks (GANs) typically suffer from overfitting when
limited training data is available. To facilitate GAN training, current methods
propose to use data-specific augmentation techniques. Despite the
effectiveness, it is difficult for these methods to scale to practical
applications. In this work, we present ScoreMix, a novel and scalable data
augmentation approach for various image synthesis tasks. We first produce
augmented samples using the convex combinations of the real samples. Then, we
optimize the augmented samples by minimizing the norms of the data scores,
i.e., the gradients of the log-density functions. This procedure enforces the
augmented samples close to the data manifold. To estimate the scores, we train
a deep estimation network with multi-scale score matching. For different image
synthesis tasks, we train the score estimation network using different data. We
do not require the tuning of the hyperparameters or modifications to the
network architecture. The ScoreMix method effectively increases the diversity
of data and reduces the overfitting problem. Moreover, it can be easily
incorporated into existing GAN models with minor modifications. Experimental
results on numerous tasks demonstrate that GAN models equipped with the
ScoreMix method achieve significant improvements.
