  Continual learning can incrementally absorb new concepts without interfering
with previously learned knowledge. Motivated by the characteristics of neural
networks, in which information is stored in weights on connections, we
investigated how to design an Innately Forgetting-Free Network (IF2Net) for
continual learning context. This study proposed a straightforward yet effective
learning paradigm by ingeniously keeping the weights relative to each seen task
untouched before and after learning a new task. We first presented the novel
representation-level learning on task sequences with random weights. This
technique refers to tweaking the drifted representations caused by
randomization back to their separate task-optimal working states, but the
involved weights are frozen and reused (opposite to well-known layer-wise
updates of weights). Then, sequential decision-making without forgetting can be
achieved by projecting the output weight updates into the parsimonious
orthogonal space, making the adaptations not disturb old knowledge while
maintaining model plasticity. IF2Net allows a single network to inherently
learn unlimited mapping rules without telling task identities at test time by
integrating the respective strengths of randomization and orthogonalization. We
validated the effectiveness of our approach in the extensive theoretical
analysis and empirical study.
