  We tackle catastrophic forgetting problem in the context of class-incremental
learning for video recognition, which has not been explored actively despite
the popularity of continual learning. Our framework addresses this challenging
task by introducing time-channel importance maps and exploiting the importance
maps for learning the representations of incoming examples via knowledge
distillation. We also incorporate a regularization scheme in our objective
function, which encourages individual features obtained from different time
steps in a video to be uncorrelated and eventually improves accuracy by
alleviating catastrophic forgetting. We evaluate the proposed approach on
brand-new splits of class-incremental action recognition benchmarks constructed
upon the UCF101, HMDB51, and Something-Something V2 datasets, and demonstrate
the effectiveness of our algorithm in comparison to the existing continual
learning methods that are originally designed for image data.
