  Continual Learning (CL) sequentially learns new tasks like human beings, with
the goal to achieve better Stability (S, remembering past tasks) and Plasticity
(P, adapting to new tasks). Due to the fact that past training data is not
available, it is valuable to explore the influence difference on S and P among
training examples, which may improve the learning pattern towards better SP.
Inspired by Influence Function (IF), we first study example influence via
adding perturbation to example weight and computing the influence derivation.
To avoid the storage and calculation burden of Hessian inverse in neural
networks, we propose a simple yet effective MetaSP algorithm to simulate the
two key steps in the computation of IF and obtain the S- and P-aware example
influence. Moreover, we propose to fuse two kinds of example influence by
solving a dual-objective optimization problem, and obtain a fused influence
towards SP Pareto optimality. The fused influence can be used to control the
update of model and optimize the storage of rehearsal. Empirical results show
that our algorithm significantly outperforms state-of-the-art methods on both
task- and class-incremental benchmark CL datasets.
