  Current state-of-the-art video models process a video clip as a long sequence
of spatio-temporal tokens. However, they do not explicitly model objects, their
interactions across the video, and instead process all the tokens in the video.
In this paper, we investigate how we can use knowledge of objects to design
better video models, namely to process fewer tokens and to improve recognition
accuracy. This is in contrast to prior works which either drop tokens at the
cost of accuracy, or increase accuracy whilst also increasing the computation
required. First, we propose an object-guided token sampling strategy that
enables us to retain a small fraction of the input tokens with minimal impact
on accuracy. And second, we propose an object-aware attention module that
enriches our feature representation with object information and improves
overall accuracy. Our resulting framework achieves better performance when
using fewer tokens than strong baselines. In particular, we match our baseline
with 30%, 40%, and 60% of the input tokens on SomethingElse,
Something-something v2, and Epic-Kitchens, respectively. When we use our model
to process the same number of tokens as our baseline, we improve by 0.6 to 4.2
points on these datasets.
