  We propose a novel framework and a solution to tackle the continual learning
(CL) problem with changing network architectures. Most CL methods focus on
adapting a single architecture to a new task/class by modifying its weights.
However, with rapid progress in architecture design, the problem of adapting
existing solutions to novel architectures becomes relevant. To address this
limitation, we propose Heterogeneous Continual Learning (HCL), where a wide
range of evolving network architectures emerge continually together with novel
data/tasks. As a solution, we build on top of the distillation family of
techniques and modify it to a new setting where a weaker model takes the role
of a teacher; meanwhile, a new stronger architecture acts as a student.
Furthermore, we consider a setup of limited access to previous data and propose
Quick Deep Inversion (QDI) to recover prior task visual features to support
knowledge transfer. QDI significantly reduces computational costs compared to
previous solutions and improves overall performance. In summary, we propose a
new setup for CL with a modified knowledge distillation paradigm and design a
quick data inversion method to enhance distillation. Our evaluation of various
benchmarks shows a significant improvement on accuracy in comparison to
state-of-the-art methods over various networks architectures.
