  Effectively modeling discriminative spatio-temporal information is essential
for segmenting activities in long action sequences. However, we observe that
existing methods are limited in weak spatio-temporal modeling capability due to
two forms of decoupled modeling: (i) cascaded interaction couples spatial and
temporal modeling, which over-smooths motion modeling over the long sequence,
and (ii) joint-shared temporal modeling adopts shared weights to model each
joint, ignoring the distinct motion patterns of different joints. We propose a
Decoupled Spatio-Temporal Framework (DeST) to address the above issues.
Firstly, we decouple the cascaded spatio-temporal interaction to avoid stacking
multiple spatio-temporal blocks, while achieving sufficient spatio-temporal
interaction. Specifically, DeST performs once unified spatial modeling and
divides the spatial features into different groups of subfeatures, which then
adaptively interact with temporal features from different layers. Since the
different sub-features contain distinct spatial semantics, the model could
learn the optimal interaction pattern at each layer. Meanwhile, inspired by the
fact that different joints move at different speeds, we propose joint-decoupled
temporal modeling, which employs independent trainable weights to capture
distinctive temporal features of each joint. On four large-scale benchmarks of
different scenes, DeST significantly outperforms current state-of-the-art
methods with less computational complexity.
