  There exists a large number of datasets for organ segmentation, which are
partially annotated and sequentially constructed. A typical dataset is
constructed at a certain time by curating medical images and annotating the
organs of interest. In other words, new datasets with annotations of new organ
categories are built over time. To unleash the potential behind these partially
labeled, sequentially-constructed datasets, we propose to incrementally learn a
multi-organ segmentation model. In each incremental learning (IL) stage, we
lose the access to previous data and annotations, whose knowledge is assumingly
captured by the current model, and gain the access to a new dataset with
annotations of new organ categories, from which we learn to update the organ
segmentation model to include the new organs. While IL is notorious for its
`catastrophic forgetting' weakness in the context of natural image analysis, we
experimentally discover that such a weakness mostly disappears for CT
multi-organ segmentation. To further stabilize the model performance across the
IL stages, we introduce a light memory module and some loss functions to
restrain the representation of different categories in feature space,
aggregating feature representation of the same class and separating feature
representation of different classes. Extensive experiments on five open-sourced
datasets are conducted to illustrate the effectiveness of our method.
