  Trajectory prediction in autonomous driving relies on accurate representation
of all relevant contexts of the driving scene, including traffic participants,
road topology, traffic signs, as well as their semantic relations to each
other. Despite increased attention to this issue, most approaches in trajectory
prediction do not consider all of these factors sufficiently. We present
SemanticFormer, an approach for predicting multimodal trajectories by reasoning
over a semantic traffic scene graph using a hybrid approach. It utilizes
high-level information in the form of meta-paths, i.e. trajectories on which an
agent is allowed to drive from a knowledge graph which is then processed by a
novel pipeline based on multiple attention mechanisms to predict accurate
trajectories. SemanticFormer comprises a hierarchical heterogeneous graph
encoder to capture spatio-temporal and relational information across agents as
well as between agents and road elements. Further, it includes a predictor to
fuse different encodings and decode trajectories with probabilities. Finally, a
refinement module assesses permitted meta-paths of trajectories and speed
profiles to obtain final predicted trajectories. Evaluation of the nuScenes
benchmark demonstrates improved performance compared to several SOTA methods.
In addition, we demonstrate that our knowledge graph can be easily added to two
graph-based existing SOTA methods, namely VectorNet and Laformer, replacing
their original homogeneous graphs. The evaluation results suggest that by
adding our knowledge graph the performance of the original methods is enhanced
by 5% and 4%, respectively.
