  Mitigating catastrophic forgetting is a key hurdle in continual learning.
Deep Generative Replay (GR) provides techniques focused on generating samples
from prior tasks to enhance the model's memory capabilities using generative AI
models ranging from Generative Adversarial Networks (GANs) to the more recent
Diffusion Models (DMs). A major issue is the deterioration in the quality of
generated data compared to the original, as the generator continuously
self-learns from its outputs. This degradation can lead to the potential risk
of catastrophic forgetting (CF) occurring in the classifier. To address this,
we propose the Gradient Projection Class-Prototype Conditional Diffusion Model
(GPPDM), a GR-based approach for continual learning that enhances image quality
in generators and thus reduces the CF in classifiers. The cornerstone of GPPDM
is a learnable class prototype that captures the core characteristics of images
in a given class. This prototype, integrated into the diffusion model's
denoising process, ensures the generation of high-quality images of the old
tasks, hence reducing the risk of CF in classifiers. Moreover, to further
mitigate the CF of diffusion models, we propose a gradient projection technique
tailored for the cross-attention layer of diffusion models to maximally
maintain and preserve the representations of old task data in the current task
as close as possible to their representations when they first arrived. Our
empirical studies on diverse datasets demonstrate that our proposed method
significantly outperforms existing state-of-the-art models, highlighting its
satisfactory ability to preserve image quality and enhance the model's memory
retention.
