  In this paper, we address the challenges posed by the substantial training
time and memory consumption associated with video transformers, focusing on the
ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
version, as our baseline for action recognition tasks. The factorised encoder
variant follows the late-fusion approach that is adopted by many state of the
art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
among the different variants of ViViT, its considerable training time and
memory requirements still pose a significant barrier to entry. Our method is
designed to lower this barrier and is based on the idea of freezing the spatial
transformer during training. This leads to a low accuracy model if naively
done. But we show that by (1) appropriately initializing the temporal
transformer (a module responsible for processing temporal information) (2)
introducing a compact adapter model connecting frozen spatial representations
((a module that selectively focuses on regions of the input image) to the
temporal transformer, we can enjoy the benefits of freezing the spatial
transformer without sacrificing accuracy. Through extensive experimentation
over 6 benchmarks, we demonstrate that our proposed training strategy
significantly reduces training costs (by $\sim 50\%$) and memory consumption
while maintaining or slightly improving performance by up to 1.79\% compared to
the baseline model. Our approach additionally unlocks the capability to utilize
larger image transformer models as our spatial transformer and access more
frames with the same memory consumption.
