  Deep reinforcement learning (RL) agents are becoming increasingly proficient
in a range of complex control tasks. However, the agent's behavior is usually
difficult to interpret due to the introduction of black-box function, making it
difficult to acquire the trust of users. Although there have been some
interesting interpretation methods for vision-based RL, most of them cannot
uncover temporal causal information, raising questions about their reliability.
To address this problem, we present a temporal-spatial causal interpretation
(TSCI) model to understand the agent's long-term behavior, which is essential
for sequential decision-making. TSCI model builds on the formulation of
temporal causality, which reflects the temporal causal relations between
sequential observations and decisions of RL agent. Then a separate causal
discovery network is employed to identify temporal-spatial causal features,
which are constrained to satisfy the temporal causality. TSCI model is
applicable to recurrent agents and can be used to discover causal features with
high efficiency once trained. The empirical results show that TSCI model can
produce high-resolution and sharp attention masks to highlight task-relevant
temporal-spatial information that constitutes most evidence about how
vision-based RL agents make sequential decisions. In addition, we further
demonstrate that our method is able to provide valuable causal interpretations
for vision-based RL agents from the temporal perspective.
