  Continual learning has recently attracted attention from the research
community, as it aims to solve long-standing limitations of classic
supervisedly-trained models. However, most research on this subject has tackled
continual learning in simple image classification scenarios. In this paper, we
present a benchmark of state-of-the-art continual learning methods on video
action recognition. Besides the increased complexity due to the temporal
dimension, the video setting imposes stronger requirements on computing
resources for top-performing rehearsal methods. To counteract the increased
memory requirements, we present two method-agnostic variants for rehearsal
methods, exploiting measures of either model confidence or data information to
select memorable samples. Our experiments show that, as expected from the
literature, rehearsal methods outperform other approaches; moreover, the
proposed memory-efficient variants are shown to be effective at retaining a
certain level of performance with a smaller buffer size.
