  In unsupervised medical image registration, the predominant approaches
involve the utilization of a encoder-decoder network architecture, allowing for
precise prediction of dense, full-resolution displacement fields from given
paired images. Despite its widespread use in the literature, we argue for the
necessity of making both the encoder and decoder learnable in such an
architecture. For this, we propose a novel network architecture, termed LessNet
in this paper, which contains only a learnable decoder, while entirely omitting
the utilization of a learnable encoder. LessNet substitutes the learnable
encoder with simple, handcrafted features, eliminating the need to learn
(optimize) network parameters in the encoder altogether. Consequently, this
leads to a compact, efficient, and decoder-only architecture for 3D medical
image registration. Evaluated on two publicly available brain MRI datasets, we
demonstrate that our decoder-only LessNet can effectively and efficiently learn
both dense displacement and diffeomorphic deformation fields in 3D.
Furthermore, our decoder-only LessNet can achieve comparable registration
performance to state-of-the-art methods such as VoxelMorph and TransMorph,
while requiring significantly fewer computational resources. Our code and
pre-trained models are available at https://github.com/xi-jia/LessNet.
