  The task of temporally detecting and segmenting actions in untrimmed videos
has seen an increased attention recently. One problem in this context arises
from the need to define and label action boundaries to create annotations for
training which is very time and cost intensive. To address this issue, we
propose an unsupervised approach for learning action classes from untrimmed
video sequences. To this end, we use a continuous temporal embedding of
framewise features to benefit from the sequential nature of activities. Based
on the latent space created by the embedding, we identify clusters of temporal
segments across all videos that correspond to semantic meaningful action
classes. The approach is evaluated on three challenging datasets, namely the
Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While
previous works assumed that the videos contain the same high level activity, we
furthermore show that the proposed approach can also be applied to a more
general setting where the content of the videos is unknown.
