  Deep reinforcement learning (DRL) provides a promising way for intelligent
agents (e.g., autonomous vehicles) to learn to navigate complex scenarios.
However, DRL with neural networks as function approximators is typically
considered a black box with little explainability and often suffers from
suboptimal performance, especially for autonomous navigation in highly
interactive multi-agent environments. To address these issues, we propose three
auxiliary tasks with spatio-temporal relational reasoning and integrate them
into the standard DRL framework, which improves the decision making performance
and provides explainable intermediate indicators. We propose to explicitly
infer the internal states (i.e., traits and intentions) of surrounding agents
(e.g., human drivers) as well as to predict their future trajectories in the
situations with and without the ego agent through counterfactual reasoning.
These auxiliary tasks provide additional supervision signals to infer the
behavior patterns of other interactive agents. Multiple variants of framework
integration strategies are compared. We also employ a spatio-temporal graph
neural network to encode relations between dynamic entities, which enhances
both internal state inference and decision making of the ego agent. Moreover,
we propose an interactivity estimation mechanism based on the difference
between predicted trajectories in these two situations, which indicates the
degree of influence of the ego agent on other agents. To validate the proposed
method, we design an intersection driving simulator based on the Intelligent
Intersection Driver Model (IIDM) that simulates vehicles and pedestrians. Our
approach achieves robust and state-of-the-art performance in terms of standard
evaluation metrics and provides explainable intermediate indicators (i.e.,
internal states, and interactivity scores) for decision making.
