  To safely and rationally participate in dense and heterogeneous traffic,
autonomous vehicles require to sufficiently analyze the motion patterns of
surrounding traffic-agents and accurately predict their future trajectories.
This is challenging because the trajectories of traffic-agents are not only
influenced by the traffic-agents themselves but also by spatial interaction
with each other. Previous methods usually rely on the sequential step-by-step
processing of Long Short-Term Memory networks (LSTMs) and merely extract the
interactions between spatial neighbors for single type traffic-agents. We
propose the Spatio-Temporal Transformer Networks (S2TNet), which models the
spatio-temporal interactions by spatio-temporal Transformer and deals with the
temporel sequences by temporal Transformer. We input additional category, shape
and heading information into our networks to handle the heterogeneity of
traffic-agents. The proposed methods outperforms state-of-the-art methods on
ApolloScape Trajectory dataset by more than 7\% on both the weighted sum of
Average and Final Displacement Error. Our code is available at
https://github.com/chenghuang66/s2tnet.
