  Multi-view action clustering leverages the complementary information from
different camera views to enhance the clustering performance. Although existing
approaches have achieved significant progress, they assume all camera views are
available in advance, which is impractical when the camera view is incremental
over time. Besides, learning the invariant information among multiple camera
views is still a challenging issue, especially in continual learning scenario.
Aiming at these problems, we propose a novel continual action clustering (CAC)
method, which is capable of learning action categories in a continual learning
manner. To be specific, we first devise a category memory library, which
captures and stores the learned categories from historical views. Then, as a
new camera view arrives, we only need to maintain a consensus partition matrix,
which can be updated by leveraging the incoming new camera view rather than
keeping all of them. Finally, a three-step alternate optimization is proposed,
in which the category memory library and consensus partition matrix are
optimized. The empirical experimental results on 6 realistic multi-view action
collections demonstrate the excellent clustering performance and time/space
efficiency of the CAC compared with 15 state-of-the-art baselines.
