  We address the problem of incremental semantic segmentation (ISS) recognizing
novel object/stuff categories continually without forgetting previous ones that
have been learned. The catastrophic forgetting problem is particularly severe
in ISS, since pixel-level ground-truth labels are available only for the novel
categories at training time. To address the problem, regularization-based
methods exploit probability calibration techniques to learn semantic
information from unlabeled pixels. While such techniques are effective, there
is still a lack of theoretical understanding of them. Replay-based methods
propose to memorize a small set of images for previous categories. They achieve
state-of-the-art performance at the cost of large memory footprint. We propose
in this paper a novel ISS method, dubbed ALIFE, that provides a better
compromise between accuracy and efficiency. To this end, we first show an
in-depth analysis on the calibration techniques to better understand the
effects on ISS. Based on this, we then introduce an adaptive logit regularizer
(ALI) that enables our model to better learn new categories, while retaining
knowledge for previous ones. We also present a feature replay scheme that
memorizes features, instead of images directly, in order to reduce memory
requirements significantly. Since a feature extractor is changed continually,
memorized features should also be updated at every incremental stage. To handle
this, we introduce category-specific rotation matrices updating the features
for each category separately. We demonstrate the effectiveness of our approach
with extensive experiments on standard ISS benchmarks, and show that our method
achieves a better trade-off in terms of accuracy and efficiency.
