  Multi-modal image registration is a challenging problem that is also an
important clinical task for many real applications and scenarios. As a first
step in analysis, deformable registration among different image modalities is
often required in order to provide complementary visual information. During
registration, semantic information is key to match homologous points and
pixels. Nevertheless, many conventional registration methods are incapable in
capturing high-level semantic anatomical dense correspondences. In this work,
we propose a novel multi-task learning system, JSSR, based on an end-to-end 3D
convolutional neural network that is composed of a generator, a registration
and a segmentation component. The system is optimized to satisfy the implicit
constraints between different tasks in an unsupervised manner. It first
synthesizes the source domain images into the target domain, then an
intra-modal registration is applied on the synthesized images and target
images. The segmentation module are then applied on the synthesized and target
images, providing additional cues based on semantic correspondences. The
supervision from another fully-annotated dataset is used to regularize the
segmentation. We extensively evaluate JSSR on a large-scale medical image
dataset containing 1,485 patient CT imaging studies of four different contrast
phases (i.e., 5,940 3D CT scans with pathological livers) on the registration,
segmentation and synthesis tasks. The performance is improved after joint
training on the registration and segmentation tasks by 0.9% and 1.9%
respectively compared to a highly competitive and accurate deep learning
baseline. The registration also consistently outperforms conventional
state-of-the-art multi-modal registration methods.
