  Explainability is essential for autonomous vehicles and other robotics
systems interacting with humans and other objects during operation. Humans need
to understand and anticipate the actions taken by the machines for trustful and
safe cooperation. In this work, we aim to develop an explainable model that
generates explanations consistent with both human domain knowledge and the
model's inherent causal relation. In particular, we focus on an essential
building block of autonomous driving, multi-agent interaction modeling. We
propose Grounded Relational Inference (GRI). It models an interactive system's
underlying dynamics by inferring an interaction graph representing the agents'
relations. We ensure a semantically meaningful interaction graph by grounding
the relational latent space into semantic interactive behaviors defined with
expert domain knowledge. We demonstrate that it can model interactive traffic
scenarios under both simulation and real-world settings, and generate semantic
graphs explaining the vehicle's behavior by their interactions.
