  Point cloud videos effectively capture real-world spatial geometries and
temporal dynamics, which are essential for enabling intelligent agents to
understand the dynamically changing 3D world we live in. Although static 3D
point cloud processing has witnessed significant advancements, designing an
effective 4D point cloud video backbone remains challenging, mainly due to the
irregular and unordered distribution of points and temporal inconsistencies
across frames. Moreover, recent state-of-the-art 4D backbones predominantly
rely on transformer-based architectures, which commonly suffer from large
computational costs due to their quadratic complexity, particularly when
processing long video sequences. To address these challenges, we propose a
novel 4D point cloud video understanding backbone based on the recently
advanced State Space Models (SSMs). Specifically, our backbone begins by
disentangling space and time in raw 4D sequences, and then establishing
spatio-temporal correlations using our newly developed Intra-frame Spatial
Mamba and Inter-frame Temporal Mamba blocks. The Intra-frame Spatial Mamba
module is designed to encode locally similar or related geometric structures
within a certain temporal searching stride, which can effectively capture
short-term dynamics. Subsequently, these locally correlated tokens are
delivered to the Inter-frame Temporal Mamba module, which globally integrates
point features across the entire video with linear complexity, further
establishing long-range motion dependencies. Experimental results on human
action recognition and 4D semantic segmentation tasks demonstrate the
superiority of our proposed method. Especially, for long video sequences, our
proposed Mamba-based method has an 87.5% GPU memory reduction, 5.36 times
speed-up, and much higher accuracy (up to +10.4%) compared with
transformer-based counterparts on MSR-Action3D dataset.
