  Catastrophic forgetting is one of the most critical challenges in Continual
Learning (CL). Recent approaches tackle this problem by projecting the gradient
update orthogonal to the gradient subspace of existing tasks. While the results
are remarkable, those approaches ignore the fact that these calculated
gradients are not guaranteed to be orthogonal to the gradient subspace of each
class due to the class deviation in tasks, e.g., distinguishing "Man" from
"Sea" v.s. differentiating "Boy" from "Girl". Therefore, this strategy may
still cause catastrophic forgetting for some classes. In this paper, we propose
Class Gradient Projection (CGP), which calculates the gradient subspace from
individual classes rather than tasks. Gradient update orthogonal to the
gradient subspace of existing classes can be effectively utilized to minimize
interference from other classes. To improve the generalization and efficiency,
we further design a Base Refining (BR) algorithm to combine similar classes and
refine class bases dynamically. Moreover, we leverage a contrastive learning
method to improve the model's ability to handle unseen tasks. Extensive
experiments on benchmark datasets demonstrate the effectiveness of our proposed
approach. It improves the previous methods by 2.0% on the CIFAR-100 dataset.
