  Recent adaptive methods for efficient video recognition mostly follow the
two-stage paradigm of "preview-then-recognition" and have achieved great
success on multiple video benchmarks. However, this two-stage paradigm involves
two visits of raw frames from coarse-grained to fine-grained during inference
(cannot be parallelized), and the captured spatiotemporal features cannot be
reused in the second stage (due to varying granularity), being not friendly to
efficiency and computation optimization. To this end, inspired by human
cognition, we propose a novel recognition paradigm of "View while Moving" for
efficient long-untrimmed video recognition. In contrast to the two-stage
paradigm, our paradigm only needs to access the raw frame once. The two phases
of coarse-grained sampling and fine-grained recognition are combined into
unified spatiotemporal modeling, showing great performance. Moreover, we
investigate the properties of semantic units in video and propose a
hierarchical mechanism to efficiently capture and reason about the unit-level
and video-level temporal semantics in long-untrimmed videos respectively.
Extensive experiments on both long-untrimmed and short-trimmed videos
demonstrate that our approach outperforms state-of-the-art methods in terms of
accuracy as well as efficiency, yielding new efficiency and accuracy trade-offs
for video spatiotemporal modeling.
