  To understand the world, we humans constantly need to relate the present to
the past, and put events in context. In this paper, we enable existing video
models to do the same. We propose a long-term feature bank---supportive
information extracted over the entire span of a video---to augment
state-of-the-art video models that otherwise would only view short clips of 2-5
seconds. Our experiments demonstrate that augmenting 3D convolutional networks
with a long-term feature bank yields state-of-the-art results on three
challenging video datasets: AVA, EPIC-Kitchens, and Charades.
