  Temporal action segmentation is a task to classify each frame in the video
with an action label. However, it is quite expensive to annotate every frame in
a large corpus of videos to construct a comprehensive supervised training
dataset. Thus in this work we propose an unsupervised method, namely SSCAP,
that operates on a corpus of unlabeled videos and predicts a likely set of
temporal segments across the videos. SSCAP leverages Self-Supervised learning
to extract distinguishable features and then applies a novel Co-occurrence
Action Parsing algorithm to not only capture the correlation among sub-actions
underlying the structure of activities, but also estimate the temporal path of
the sub-actions in an accurate and general way. We evaluate on both classic
datasets (Breakfast, 50Salads) and the emerging fine-grained action dataset
(FineGym) with more complex activity structures and similar sub-actions.
Results show that SSCAP achieves state-of-the-art performance on all datasets
and can even outperform some weakly-supervised approaches, demonstrating its
effectiveness and generalizability.
