  Unsupervised image registration commonly adopts U-Net style networks to
predict dense displacement fields in the full-resolution spatial domain. For
high-resolution volumetric image data, this process is however
resource-intensive and time-consuming. To tackle this problem, we propose the
Fourier-Net, replacing the expansive path in a U-Net style network with a
parameter-free model-driven decoder. Specifically, instead of our Fourier-Net
learning to output a full-resolution displacement field in the spatial domain,
we learn its low-dimensional representation in a band-limited Fourier domain.
This representation is then decoded by our devised model-driven decoder
(consisting of a zero padding layer and an inverse discrete Fourier transform
layer) to the dense, full-resolution displacement field in the spatial domain.
These changes allow our unsupervised Fourier-Net to contain fewer parameters
and computational operations, resulting in faster inference speeds. Fourier-Net
is then evaluated on two public 3D brain datasets against various
state-of-the-art approaches. For example, when compared to a recent
transformer-based method, named TransMorph, our Fourier-Net, which only uses
2.2\% of its parameters and 6.66\% of the multiply-add operations, achieves a
0.5\% higher Dice score and an 11.48 times faster inference speed. Code is
available at \url{https://github.com/xi-jia/Fourier-Net}.
