  Incremental learning of semantic segmentation has emerged as a promising
strategy for visual scene interpretation in the open- world setting. However,
it remains challenging to acquire novel classes in an online fashion for the
segmentation task, mainly due to its continuously-evolving semantic label
space, partial pixelwise ground-truth annotations, and constrained data
availability. To ad- dress this, we propose an incremental learning strategy
that can fast adapt deep segmentation models without catastrophic forgetting,
using a streaming input data with pixel annotations on the novel classes only.
To this end, we develop a uni ed learning strategy based on the
Expectation-Maximization (EM) framework, which integrates an iterative
relabeling strategy that lls in the missing labels and a rehearsal-based
incremental learning step that balances the stability-plasticity of the model.
Moreover, our EM algorithm adopts an adaptive sampling method to select
informative train- ing data and a class-balancing training strategy in the
incremental model updates, both improving the e cacy of model learning. We
validate our approach on the PASCAL VOC 2012 and ADE20K datasets, and the
results demonstrate its superior performance over the existing incremental
methods.
