  Action Quality Assessment (AQA) is a task that tries to answer how well an
action is carried out. While remarkable progress has been achieved, existing
works on AQA assume that all the training data are visible for training at one
time, but do not enable continual learning on assessing new technical actions.
In this work, we address such a Continual Learning problem in AQA
(Continual-AQA), which urges a unified model to learn AQA tasks sequentially
without forgetting. Our idea for modeling Continual-AQA is to sequentially
learn a task-consistent score-discriminative feature distribution, in which the
latent features express a strong correlation with the score labels regardless
of the task or action types.From this perspective, we aim to mitigate the
forgetting in Continual-AQA from two aspects. Firstly, to fuse the features of
new and previous data into a score-discriminative distribution, a novel
Feature-Score Correlation-Aware Rehearsal is proposed to store and reuse data
from previous tasks with limited memory size. Secondly, an Action
General-Specific Graph is developed to learn and decouple the action-general
and action-specific knowledge so that the task-consistent score-discriminative
features can be better extracted across various tasks. Extensive experiments
are conducted to evaluate the contributions of proposed components. The
comparisons with the existing continual learning methods additionally verify
the effectiveness and versatility of our approach. Data and code are available
at https://github.com/iSEE-Laboratory/Continual-AQA.
