  Vision Transformers have achieved impressive performance in video
classification, while suffering from the quadratic complexity caused by the
Softmax attention mechanism. Some studies alleviate the computational costs by
reducing the number of tokens in attention calculation, but the complexity is
still quadratic. Another promising way is to replace Softmax attention with
linear attention, which owns linear complexity but presents a clear performance
drop. We find that such a drop in linear attention results from the lack of
attention concentration on critical features. Therefore, we propose a feature
fixation module to reweight the feature importance of the query and key before
computing linear attention. Specifically, we regard the query, key, and value
as various latent representations of the input token, and learn the feature
fixation ratio by aggregating Query-Key-Value information. This is beneficial
for measuring the feature importance comprehensively. Furthermore, we enhance
the feature fixation by neighborhood association, which leverages additional
guidance from spatial and temporal neighbouring tokens. The proposed method
significantly improves the linear attention baseline and achieves
state-of-the-art performance among linear video Transformers on three popular
video classification benchmarks. With fewer parameters and higher efficiency,
our performance is even comparable to some Softmax-based quadratic
Transformers.
