  We propose to forecast future hand-object interactions given an egocentric
video. Instead of predicting action labels or pixels, we directly predict the
hand motion trajectory and the future contact points on the next active object
(i.e., interaction hotspots). This relatively low-dimensional representation
provides a concrete description of future interactions. To tackle this task, we
first provide an automatic way to collect trajectory and hotspots labels on
large-scale data. We then use this data to train an Object-Centric Transformer
(OCT) model for prediction. Our model performs hand and object interaction
reasoning via the self-attention mechanism in Transformers. OCT also provides a
probabilistic framework to sample the future trajectory and hotspots to handle
uncertainty in prediction. We perform experiments on the Epic-Kitchens-55,
Epic-Kitchens-100, and EGTEA Gaze+ datasets, and show that OCT significantly
outperforms state-of-the-art approaches by a large margin. Project page is
available at https://stevenlsw.github.io/hoi-forecast .
