  Whole slide image (WSI) analysis has become increasingly important in the
medical imaging community, enabling automated and objective diagnosis,
prognosis, and therapeutic-response prediction. However, in clinical practice,
the ever-evolving environment hamper the utility of WSI analysis models. In
this paper, we propose the FIRST continual learning framework for WSI analysis,
named ConSlide, to tackle the challenges of enormous image size, utilization of
hierarchical structure, and catastrophic forgetting by progressive model
updating on multiple sequential datasets. Our framework contains three key
components. The Hierarchical Interaction Transformer (HIT) is proposed to model
and utilize the hierarchical structural knowledge of WSI. The
Breakup-Reorganize (BuRo) rehearsal method is developed for WSI data replay
with efficient region storing buffer and WSI reorganizing operation. The
asynchronous updating mechanism is devised to encourage the network to learn
generic and specific knowledge respectively during the replay stage, based on a
nested cross-scale similarity learning (CSSL) module. We evaluated the proposed
ConSlide on four public WSI datasets from TCGA projects. It performs best over
other state-of-the-art methods with a fair WSI-based continual learning setting
and achieves a better trade-off of the overall performance and forgetting on
previous task
