  Understanding the structure of complex activities in untrimmed videos is a
challenging task in the area of action recognition. One problem here is that
this task usually requires a large amount of hand-annotated minute- or even
hour-long video data, but annotating such data is very time consuming and can
not easily be automated or scaled. To address this problem, this paper proposes
an approach for the unsupervised learning of actions in untrimmed video
sequences based on a joint visual-temporal embedding space. To this end, we
combine a visual embedding based on a predictive U-Net architecture with a
temporal continuous function. The resulting representation space allows
detecting relevant action clusters based on their visual as well as their
temporal appearance. The proposed method is evaluated on three standard
benchmark datasets, Breakfast Actions, INRIA YouTube Instructional Videos, and
50 Salads. We show that the proposed approach is able to provide a meaningful
visual and temporal embedding out of the visual cues present in contiguous
video frames and is suitable for the task of unsupervised temporal segmentation
of actions.
