  Self-supervised video representation methods typically focus on the
representation of temporal attributes in videos. However, the role of
stationary versus non-stationary attributes is less explored: Stationary
features, which remain similar throughout the video, enable the prediction of
video-level action classes. Non-stationary features, which represent temporally
varying attributes, are more beneficial for downstream tasks involving more
fine-grained temporal understanding, such as action segmentation. We argue that
a single representation to capture both types of features is sub-optimal, and
propose to decompose the representation space into stationary and
non-stationary features via contrastive learning from long and short views,
i.e. long video sequences and their shorter sub-sequences. Stationary features
are shared between the short and long views, while non-stationary features
aggregate the short views to match the corresponding long view. To empirically
verify our approach, we demonstrate that our stationary features work
particularly well on an action recognition downstream task, while our
non-stationary features perform better on action segmentation. Furthermore, we
analyse the learned representations and find that stationary features capture
more temporally stable, static attributes, while non-stationary features
encompass more temporally varying ones.
