  Recently, video transformers have shown great success in video understanding,
exceeding CNN performance; yet existing video transformer models do not
explicitly model objects, although objects can be essential for recognizing
actions. In this work, we present Object-Region Video Transformers (ORViT), an
\emph{object-centric} approach that extends video transformer layers with a
block that directly incorporates object representations. The key idea is to
fuse object-centric representations starting from early layers and propagate
them into the transformer-layers, thus affecting the spatio-temporal
representations throughout the network. Our ORViT block consists of two
object-level streams: appearance and dynamics. In the appearance stream, an
"Object-Region Attention" module applies self-attention over the patches and
\emph{object regions}. In this way, visual object regions interact with uniform
patch tokens and enrich them with contextualized object information. We further
model object dynamics via a separate "Object-Dynamics Module", which captures
trajectory interactions, and show how to integrate the two streams. We evaluate
our model on four tasks and five datasets: compositional and few-shot action
recognition on SomethingElse, spatio-temporal action detection on AVA, and
standard action recognition on Something-Something V2, Diving48 and
Epic-Kitchen100. We show strong performance improvement across all tasks and
datasets considered, demonstrating the value of a model that incorporates
object representations into a transformer architecture. For code and pretrained
models, visit the project page at \url{https://roeiherz.github.io/ORViT/}
