  For human action understanding, a popular research direction is to analyze
short video clips with unambiguous semantic content, such as jumping and
drinking. However, methods for understanding short semantic actions cannot be
directly translated to long human dynamics such as dancing, where it becomes
challenging even to label the human movements semantically. Meanwhile, the
natural language processing (NLP) community has made progress in solving a
similar challenge of annotation scarcity by large-scale pre-training, which
improves several downstream tasks with one model. In this work, we study how to
segment and cluster videos into recurring temporal patterns in a
self-supervised way, namely acton discovery, the main roadblock towards video
tokenization. We propose a two-stage framework that first obtains a frame-wise
representation by contrasting two augmented views of video frames conditioned
on their temporal context. The frame-wise representations across a collection
of videos are then clustered by K-means. Actons are then automatically
extracted by forming a continuous motion sequence from frames within the same
cluster. We evaluate the frame-wise representation learning step by Kendall's
Tau and the lexicon building step by normalized mutual information and language
entropy. We also study three applications of this tokenization: genre
classification, action segmentation, and action composition. On the AIST++ and
PKU-MMD datasets, actons bring significant performance improvements compared to
several baselines.
