  Existing convolutional neural networks widely adopt spatial down-/up-sampling
for multi-scale modeling. However, spatial up-sampling operators (\emph{e.g.},
interpolation, transposed convolution, and un-pooling) heavily depend on local
pixel attention, incapably exploring the global dependency. In contrast, the
Fourier domain obeys the nature of global modeling according to the spectral
convolution theorem. Unlike the spatial domain that performs up-sampling with
the property of local similarity, up-sampling in the Fourier domain is more
challenging as it does not follow such a local property. In this study, we
propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve
these issues. We revisit the relationships between spatial and Fourier domains
and reveal the transform rules on the features of different resolutions in the
Fourier domain, which provide key insights for FourierUp's designs. FourierUp
as a generic operator consists of three key components: 2D discrete Fourier
transform, Fourier dimension increase rules, and 2D inverse Fourier transform,
which can be directly integrated with existing networks. Extensive experiments
across multiple computer vision tasks, including object detection, image
segmentation, image de-raining, image dehazing, and guided image
super-resolution, demonstrate the consistent performance gains obtained by
introducing our FourierUp.
