  Semi-supervised temporal action segmentation (SS-TA) aims to perform
frame-wise classification in long untrimmed videos, where only a fraction of
videos in the training set have labels. Recent studies have shown the potential
of contrastive learning in unsupervised representation learning using
unlabelled data. However, learning the representation of each frame by
unsupervised contrastive learning for action segmentation remains an open and
challenging problem. In this paper, we propose a novel Semantic-guided
Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit
(SMC-NCA) to extract strong frame-wise representations for SS-TAS.
Specifically, for representation learning, SMC is first used to explore intra-
and inter-information variations in a unified and contrastive way, based on
action-specific semantic information and temporal information highlighting
relations between actions. Then, the NCA module, which is responsible for
enforcing spatial consistency between neighbourhoods centered at different
frames to alleviate over-segmentation issues, works alongside SMC for
semi-supervised learning (SSL). Our SMC outperforms the other state-of-the-art
methods on three benchmarks, offering improvements of up to 17.8% and 12.6% in
terms of Edit distance and accuracy, respectively. Additionally, the NCA unit
results in significantly better segmentation performance in the presence of
only 5% labelled videos. We also demonstrate the generalizability and
effectiveness of the proposed method on our Parkinson Disease's Mouse Behaviour
(PDMB) dataset. Code is available at https://github.com/FeixiangZhou/SMC-NCA.
