  The ability to learn sequentially from different data sites is crucial for a
deep network in solving practical medical image diagnosis problems due to
privacy restrictions and storage limitations. However, adapting on incoming
site leads to catastrophic forgetting on past sites and decreases
generalizablity on unseen sites. Existing Continual Learning (CL) and Domain
Generalization (DG) methods have been proposed to solve these two challenges
respectively, but none of them can address both simultaneously. Recognizing
this limitation, this paper proposes a novel training paradigm, learning
towards Synchronous Memorizability and Generalizability (SMG-Learning). To
achieve this, we create the orientational gradient alignment to ensure
memorizability on previous sites, and arbitrary gradient alignment to enhance
generalizability on unseen sites. This approach is named as Parallel Gradient
Alignment (PGA). Furthermore, we approximate the PGA as dual meta-objectives
using the first-order Taylor expansion to reduce computational cost of aligning
gradients. Considering that performing gradient alignments, especially for
previous sites, is not feasible due to the privacy constraints, we design a
Site-Modulated Diffusion (SMD) model to generate images with site-specific
learnable prompts, replaying images have similar data distributions as previous
sites. We evaluate our method on two medical image segmentation tasks, where
data from different sites arrive sequentially. Experimental results show that
our method efficiently enhances both memorizability and generalizablity better
than other state-of-the-art methods, delivering satisfactory performance across
all sites. Our code will be available at:
https://github.com/dyxu-cuhkcse/SMG-Learning.
