  Several recent studies advocate the use of spectral discriminators, which
evaluate the Fourier spectra of images for generative modeling. However, the
effectiveness of the spectral discriminators is not well interpreted yet. We
tackle this issue by examining the spectral discriminators in the context of
perceptual image super-resolution (i.e., GAN-based SR), as SR image quality is
susceptible to spectral changes. Our analyses reveal that the spectral
discriminator indeed performs better than the ordinary (a.k.a. spatial)
discriminator in identifying the differences in the high-frequency range;
however, the spatial discriminator holds an advantage in the low-frequency
range. Thus, we suggest that the spectral and spatial discriminators shall be
used simultaneously. Moreover, we improve the spectral discriminators by first
calculating the patch-wise Fourier spectrum and then aggregating the spectra by
Transformer. We verify the effectiveness of the proposed method twofold. On the
one hand, thanks to the additional spectral discriminator, our obtained SR
images have their spectra better aligned to those of the real images, which
leads to a better PD tradeoff. On the other hand, our ensembled discriminator
predicts the perceptual quality more accurately, as evidenced in the
no-reference image quality assessment task.
