  Transformer becomes prevalent in computer vision, especially for high-level
vision tasks. However, adopting Transformer in the generative adversarial
network (GAN) framework is still an open yet challenging problem. In this
paper, we conduct a comprehensive empirical study to investigate the properties
of Transformer in GAN for high-fidelity image synthesis. Our analysis
highlights and reaffirms the importance of feature locality in image
generation, although the merits of the locality are well known in the
classification task. Perhaps more interestingly, we find the residual
connections in self-attention layers harmful for learning Transformer-based
discriminators and conditional generators. We carefully examine the influence
and propose effective ways to mitigate the negative impacts. Our study leads to
a new alternative design of Transformers in GAN, a convolutional neural network
(CNN)-free generator termed as STrans-G, which achieves competitive results in
both unconditional and conditional image generations. The Transformer-based
discriminator, STrans-D, also significantly reduces its gap against the
CNN-based discriminators.
