  Deep-learning based Super-Resolution (SR) methods have exhibited promising
performance under non-blind setting where blur kernel is known. However, blur
kernels of Low-Resolution (LR) images in different practical applications are
usually unknown. It may lead to significant performance drop when degradation
process of training images deviates from that of real images. In this paper, we
propose a novel blind SR framework to super-resolve LR images degraded by
arbitrary blur kernel with accurate kernel estimation in frequency domain. To
our best knowledge, this is the first deep learning method which conducts blur
kernel estimation in frequency domain. Specifically, we first demonstrate that
feature representation in frequency domain is more conducive for blur kernel
reconstruction than in spatial domain. Next, we present a Spectrum-to-Kernel
(S$2$K) network to estimate general blur kernels in diverse forms. We use a
Conditional GAN (CGAN) combined with SR-oriented optimization target to learn
the end-to-end translation from degraded images' spectra to unknown kernels.
Extensive experiments on both synthetic and real-world images demonstrate that
our proposed method sufficiently reduces blur kernel estimation error, thus
enables the off-the-shelf non-blind SR methods to work under blind setting
effectively, and achieves superior performance over state-of-the-art blind SR
methods, averagely by 1.39dB, 0.48dB on commom blind SR setting (with Gaussian
kernels) for scales $2\times$ and $4\times$, respectively.
