  In this paper, we tackle the problem of detecting objects in 3D and
forecasting their future motion in the context of self-driving. Towards this
goal, we design a novel approach that explicitly takes into account the
interactions between actors. To capture their spatial-temporal dependencies, we
propose a recurrent neural network with a novel Transformer architecture, which
we call the Interaction Transformer. Importantly, our model can be trained
end-to-end, and runs in real-time. We validate our approach on two challenging
real-world datasets: ATG4D and nuScenes. We show that our approach can
outperform the state-of-the-art on both datasets. In particular, we
significantly improve the social compliance between the estimated future
trajectories, resulting in far fewer collisions between the predicted actors.
