  Generative Adversarial Networks (GANs) are the most popular image generation
models that have achieved remarkable progress on various computer vision tasks.
However, training instability is still one of the open problems for all
GAN-based algorithms. Quite a number of methods have been proposed to stabilize
the training of GANs, the focuses of which were respectively put on the loss
functions, regularization and normalization technologies, training algorithms,
and model architectures. Different from the above methods, in this paper, a new
perspective on stabilizing GANs training is presented. It is found that
sometimes the images produced by the generator act like adversarial examples of
the discriminator during the training process, which may be part of the reason
causing the unstable training of GANs. With this finding, we propose the Direct
Adversarial Training (DAT) method to stabilize the training process of GANs.
Furthermore, we prove that the DAT method is able to minimize the Lipschitz
constant of the discriminator adaptively. The advanced performance of DAT is
verified on multiple loss functions, network architectures, hyper-parameters,
and datasets. Specifically, DAT achieves significant improvements of 11.5% FID
on CIFAR-100 unconditional generation based on SSGAN, 10.5% FID on STL-10
unconditional generation based on SSGAN, and 13.2% FID on LSUN-Bedroom
unconditional generation based on SSGAN. Code will be available at
https://github.com/iceli1007/DAT-GAN
