  Recently, joint registration and segmentation has been formulated in a deep
learning setting, by the definition of joint loss functions. In this work, we
investigate joining these tasks at the architectural level. We propose a
registration network that integrates segmentation propagation between images,
and a segmentation network to predict the segmentation directly. These networks
are connected into a single joint architecture via so-called cross-stitch
units, allowing information to be exchanged between the tasks in a learnable
manner. The proposed method is evaluated in the context of adaptive
image-guided radiotherapy, using daily prostate CT imaging. Two datasets from
different institutes and manufacturers were involved in the study. The first
dataset was used for training (12 patients) and validation (6 patients), while
the second dataset was used as an independent test set (14 patients). In terms
of mean surface distance, our approach achieved $1.06 \pm 0.3$ mm, $0.91 \pm
0.4$ mm, $1.27 \pm 0.4$ mm, and $1.76 \pm 0.8$ mm on the validation set and
$1.82 \pm 2.4$ mm, $2.45 \pm 2.4$ mm, $2.45 \pm 5.0$ mm, and $2.57 \pm 2.3$ mm
on the test set for the prostate, bladder, seminal vesicles, and rectum,
respectively. The proposed multi-task network outperformed single-task
networks, as well as a network only joined through the loss function, thus
demonstrating the capability to leverage the individual strengths of the
segmentation and registration tasks. The obtained performance as well as the
inference speed make this a promising candidate for daily re-contouring in
adaptive radiotherapy, potentially reducing treatment-related side effects and
improving quality-of-life after treatment.
