  Deep learning based single image super-resolution models have been widely
studied and superb results are achieved in upscaling low-resolution images with
fixed scale factor and downscaling degradation kernel. To improve real world
applicability of such models, there are growing interests to develop models
optimized for arbitrary upscaling factors. Our proposed method is the first to
treat arbitrary rescaling, both upscaling and downscaling, as one unified
process. Using joint optimization of both directions, the proposed model is
able to learn upscaling and downscaling simultaneously and achieve
bidirectional arbitrary image rescaling. It improves the performance of current
arbitrary upscaling models by a large margin while at the same time learns to
maintain visual perception quality in downscaled images. The proposed model is
further shown to be robust in cycle idempotence test, free of severe
degradations in reconstruction accuracy when the downscaling-to-upscaling cycle
is applied repetitively. This robustness is beneficial for image rescaling in
the wild when this cycle could be applied to one image for multiple times. It
also performs well on tests with arbitrary large scales and asymmetric scales,
even when the model is not trained with such tasks. Extensive experiments are
conducted to demonstrate the superior performance of our model.
