  Accurately predicting the possible behaviors of traffic participants is an
essential capability for autonomous vehicles. Since autonomous vehicles need to
navigate in dynamically changing environments, they are expected to make
accurate predictions regardless of where they are and what driving
circumstances they encountered. Several methodologies have been proposed to
solve prediction problems under different traffic situations. These works
usually combine agent trajectories with either color-coded or vectorized high
definition (HD) map as input representations and encode this information for
behavior prediction tasks. However, not all the information is relevant in the
scene for the forecasting and such irrelevant information may be even
distracting to the forecasting in certain situations. Therefore, in this paper,
we propose a novel generic representation for various driving environments by
taking the advantage of semantics and domain knowledge. Using semantics enables
situations to be modeled in a uniform way and applying domain knowledge filters
out unrelated elements to target vehicle's future behaviors. We then propose a
general semantic behavior prediction framework to effectively utilize these
representations by formulating them into spatial-temporal semantic graphs and
reasoning internal relations among these graphs. We theoretically and
empirically validate the proposed framework under highly interactive and
complex scenarios, demonstrating that our method not only achieves
state-of-the-art performance, but also processes desirable zero-shot
transferability.
