  Many medical datasets have recently been created for medical image
segmentation tasks, and it is natural to question whether we can use them to
sequentially train a single model that (1) performs better on all these
datasets, and (2) generalizes well and transfers better to the unknown target
site domain. Prior works have achieved this goal by jointly training one model
on multi-site datasets, which achieve competitive performance on average but
such methods rely on the assumption about the availability of all training
data, thus limiting its effectiveness in practical deployment. In this paper,
we propose a novel multi-site segmentation framework called
incremental-transfer learning (ITL), which learns a model from multi-site
datasets in an end-to-end sequential fashion. Specifically, "incremental"
refers to training sequentially constructed datasets, and "transfer" is
achieved by leveraging useful information from the linear combination of
embedding features on each dataset. In addition, we introduce our ITL
framework, where we train the network including a site-agnostic encoder with
pre-trained weights and at most two segmentation decoder heads. We also design
a novel site-level incremental loss in order to generalize well on the target
domain. Second, we show for the first time that leveraging our ITL training
scheme is able to alleviate challenging catastrophic forgetting problems in
incremental learning. We conduct experiments using five challenging benchmark
datasets to validate the effectiveness of our incremental-transfer learning
approach. Our approach makes minimal assumptions on computation resources and
domain-specific expertise, and hence constitutes a strong starting point in
multi-site medical image segmentation.
