  Continual learning (CL) remains a significant challenge for deep neural
networks, as it is prone to forgetting previously acquired knowledge. Several
approaches have been proposed in the literature, such as experience rehearsal,
regularization, and parameter isolation, to address this problem. Although
almost zero forgetting can be achieved in task-incremental learning,
class-incremental learning remains highly challenging due to the problem of
inter-task class separation. Limited access to previous task data makes it
difficult to discriminate between classes of current and previous tasks. To
address this issue, we propose `Attention-Guided Incremental Learning' (AGILE),
a novel rehearsal-based CL approach that incorporates compact task attention to
effectively reduce interference between tasks. AGILE utilizes lightweight,
learnable task projection vectors to transform the latent representations of a
shared task attention module toward task distribution. Through extensive
empirical evaluation, we show that AGILE significantly improves generalization
performance by mitigating task interference and outperforming rehearsal-based
approaches in several CL scenarios. Furthermore, AGILE can scale well to a
large number of tasks with minimal overhead while remaining well-calibrated
with reduced task-recency bias.
