  Most image super-resolution (SR) methods are developed on synthetic
low-resolution (LR) and high-resolution (HR) image pairs that are constructed
by a predetermined operation, e.g., bicubic downsampling. As existing methods
typically learn an inverse mapping of the specific function, they produce
blurry results when applied to real-world images whose exact formulation is
different and unknown. Therefore, several methods attempt to synthesize much
more diverse LR samples or learn a realistic downsampling model. However, due
to restrictive assumptions on the downsampling process, they are still biased
and less generalizable. This study proposes a novel method to simulate an
unknown downsampling process without imposing restrictive prior knowledge. We
propose a generalizable low-frequency loss (LFL) in the adversarial training
framework to imitate the distribution of target LR images without using any
paired examples. Furthermore, we design an adaptive data loss (ADL) for the
downsampler, which can be adaptively learned and updated from the data during
the training loops. Extensive experiments validate that our downsampling model
can facilitate existing SR methods to perform more accurate reconstructions on
various synthetic and real-world examples than the conventional approaches.
