  Transformers have achieved the state-of-the-art performance on solving the
inverse problem of Snapshot Compressive Imaging (SCI) for video, whose
ill-posedness is rooted in the mixed degradation of spatial masking and
temporal aliasing. However, previous Transformers lack an insight into the
degradation and thus have limited performance and efficiency. In this work, we
tailor an efficient reconstruction architecture without temporal aggregation in
early layers and Hierarchical Separable Video Transformer (HiSViT) as building
block. HiSViT is built by multiple groups of Cross-Scale Separable Multi-head
Self-Attention (CSS-MSA) and Gated Self-Modulated Feed-Forward Network
(GSM-FFN) with dense connections, each of which is conducted within a separate
channel portions at a different scale, for multi-scale interactions and
long-range modeling. By separating spatial operations from temporal ones,
CSS-MSA introduces an inductive bias of paying more attention within frames
instead of between frames while saving computational overheads. GSM-FFN further
enhances the locality via gated mechanism and factorized spatial-temporal
convolutions. Extensive experiments demonstrate that our method outperforms
previous methods by $\!>\!0.5$ dB with comparable or fewer parameters and
complexity. The source codes and pretrained models are released at
https://github.com/pwangcs/HiSViT.
