  Recent action recognition models have achieved impressive results by
integrating objects, their locations and interactions. However, obtaining dense
structured annotations for each frame is tedious and time-consuming, making
these methods expensive to train and less scalable. At the same time, if a
small set of annotated images is available, either within or outside the domain
of interest, how could we leverage these for a video downstream task? We
propose a learning framework StructureViT (SViT for short), which demonstrates
how utilizing the structure of a small number of images only available during
training can improve a video model. SViT relies on two key insights. First, as
both images and videos contain structured information, we enrich a transformer
model with a set of \emph{object tokens} that can be used across images and
videos. Second, the scene representations of individual frames in video should
"align" with those of still images. This is achieved via a \emph{Frame-Clip
Consistency} loss, which ensures the flow of structured information between
images and videos. We explore a particular instantiation of scene structure,
namely a \emph{Hand-Object Graph}, consisting of hands and objects with their
locations as nodes, and physical relations of contact/no-contact as edges. SViT
shows strong performance improvements on multiple video understanding tasks and
datasets. Furthermore, it won in the Ego4D CVPR'22 Object State Localization
challenge. For code and pretrained models, visit the project page at
\url{https://eladb3.github.io/SViT/}
