  Continual Semantic Segmentation (CSS) extends static semantic segmentation by
incrementally introducing new classes for training. To alleviate the
catastrophic forgetting issue in CSS, a memory buffer that stores a small
number of samples from the previous classes is constructed for replay. However,
existing methods select the memory samples either randomly or based on a
single-factor-driven handcrafted strategy, which has no guarantee to be
optimal. In this work, we propose a novel memory sample selection mechanism
that selects informative samples for effective replay in a fully automatic way
by considering comprehensive factors including sample diversity and class
performance. Our mechanism regards the selection operation as a decision-making
process and learns an optimal selection policy that directly maximizes the
validation performance on a reward set. To facilitate the selection decision,
we design a novel state representation and a dual-stage action space. Our
extensive experiments on Pascal-VOC 2012 and ADE 20K datasets demonstrate the
effectiveness of our approach with state-of-the-art (SOTA) performance
achieved, outperforming the second-place one by 12.54% for the 6stage setting
on Pascal-VOC 2012.
