  This paper presents an unsupervised transformer-based framework for temporal
activity segmentation which leverages not only frame-level cues but also
segment-level cues. This is in contrast with previous methods which often rely
on frame-level information only. Our approach begins with a frame-level
prediction module which estimates framewise action classes via a transformer
encoder. The frame-level prediction module is trained in an unsupervised manner
via temporal optimal transport. To exploit segment-level information, we
utilize a segment-level prediction module and a frame-to-segment alignment
module. The former includes a transformer decoder for estimating video
transcripts, while the latter matches frame-level features with segment-level
features, yielding permutation-aware segmentation results. Moreover, inspired
by temporal optimal transport, we introduce simple-yet-effective pseudo labels
for unsupervised training of the above modules. Our experiments on four public
datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop
Assembly show that our approach achieves comparable or better performance than
previous methods in unsupervised activity segmentation.
