  This paper considers the problem of spatiotemporal object-centric reasoning
in videos. Central to our approach is the notion of object permanence, i.e.,
the ability to reason about the location of objects as they move through the
video while being occluded, contained or carried by other objects. Existing
deep learning based approaches often suffer from spatiotemporal biases when
applied to video reasoning problems. We propose Hopper, which uses a Multi-hop
Transformer for reasoning object permanence in videos. Given a video and a
localization query, Hopper reasons over image and object tracks to
automatically hop over critical frames in an iterative fashion to predict the
final position of the object of interest. We demonstrate the effectiveness of
using a contrastive loss to reduce spatiotemporal biases. We evaluate over
CATER dataset and find that Hopper achieves 73.2% Top-1 accuracy using just 1
FPS by hopping through just a few critical frames. We also demonstrate Hopper
can perform long-term reasoning by building a CATER-h dataset that requires
multi-step reasoning to localize objects of interest correctly.
