  Lifelong learning without catastrophic forgetting (i.e., resiliency) remains
an open problem for deep neural networks. The prior art mostly focuses on
convolutional neural networks. With the increasing dominance of Transformers in
deep learning, it is a pressing need to study lifelong learning with
Transformers. Due to the complexity of training Transformers in practice, for
lifelong learning, a question naturally arises: Can Transformers be learned to
grow in a task aware way, that is to be dynamically transformed by introducing
lightweight learnable plastic components to the architecture, while retaining
the parameter-heavy, but stable components at streaming tasks? To that end,
motivated by the lifelong learning capability maintained by the functionality
of Hippocampi in human brain, we explore what would be, and how to implement,
Artificial Hippocampi (ArtiHippo) in Transformers. We present a method to
identify, and learn to grow, ArtiHippo in Vision Transformers (ViTs) for
resilient lifelong learning in four aspects: (i) Where to place ArtiHippo to
enable plasticity while preserving the core function of ViTs at streaming
tasks? (ii) How to represent and realize ArtiHippo to ensure expressivity and
adaptivity for tackling tasks of different nature in lifelong learning? (iii)
How to learn to grow ArtiHippo to exploit task synergies (i.e., the learned
knowledge) and overcome catastrophic forgetting? (iv) How to harness the best
of our proposed ArtiHippo and prompting-based approaches? In experiments, we
test the proposed method on the challenging Visual Domain Decathlon (VDD)
benchmark and the 5-Dataset benchmark under the task-incremental lifelong
learning setting. It obtains consistently better performance than the prior art
with sensible ArtiHippo learned continually. To our knowledge, it is the first
attempt of lifelong learning with ViTs on the challenging VDD benchmark.
