  Endoscopy is a widely used technique for the early detection of diseases or
robotic-assisted minimally invasive surgery (RMIS). Numerous deep learning
(DL)-based research works have been developed for automated diagnosis or
processing of endoscopic view. However, existing DL models may suffer from
catastrophic forgetting. When new target classes are introduced over time or
cross institutions, the performance of old classes may suffer severe
degradation. More seriously, data privacy and storage issues may lead to the
unavailability of old data when updating the model. Therefore, it is necessary
to develop a continual learning (CL) methodology to solve the problem of
catastrophic forgetting in endoscopic image segmentation. To tackle this, we
propose a Endoscopy Continual Semantic Segmentation (EndoCSS) framework that
does not involve the storage and privacy issues of exemplar data. The framework
includes a mini-batch pseudo-replay (MB-PR) mechanism and a self-adaptive noisy
cross-entropy (SAN-CE) loss. The MB-PR strategy circumvents privacy and storage
issues by generating pseudo-replay images through a generative model.
Meanwhile, the MB-PR strategy can also correct the model deviation to the
replay data and current training data, which is aroused by the significant
difference in the amount of current and replay images. Therefore, the model can
perform effective representation learning on both new and old tasks. SAN-CE
loss can help model fitting by adjusting the model's output logits, and also
improve the robustness of training. Extensive continual semantic segmentation
(CSS) experiments on public datasets demonstrate that our method can robustly
and effectively address the catastrophic forgetting brought by class increment
in endoscopy scenes. The results show that our framework holds excellent
potential for real-world deployment in a streaming learning manner.
