  The real-world deployment of an autonomous driving system requires its
components to run on-board and in real-time, including the motion prediction
module that predicts the future trajectories of surrounding traffic
participants. Existing agent-centric methods have demonstrated outstanding
performance on public benchmarks. However, they suffer from high computational
overhead and poor scalability as the number of agents to be predicted
increases. To address this problem, we introduce the K-nearest neighbor
attention with relative pose encoding (KNARPE), a novel attention mechanism
allowing the pairwise-relative representation to be used by Transformers. Then,
based on KNARPE we present the Heterogeneous Polyline Transformer with Relative
pose encoding (HPTR), a hierarchical framework enabling asynchronous token
update during the online inference. By sharing contexts among agents and
reusing the unchanged contexts, our approach is as efficient as scene-centric
methods, while performing on par with state-of-the-art agent-centric methods.
Experiments on Waymo and Argoverse-2 datasets show that HPTR achieves superior
performance among end-to-end methods that do not apply expensive
post-processing or model ensembling. The code is available at
https://github.com/zhejz/HPTR.
