  Explainability plays an increasingly important role in machine learning.
Furthermore, humans view the world through a causal lens and thus prefer causal
explanations over associational ones. Therefore, in this paper, we develop a
causal explanation mechanism that quantifies the causal importance of states on
actions and such importance over time. We also demonstrate the advantages of
our mechanism over state-of-the-art associational methods in terms of RL policy
explanation through a series of simulation studies, including crop irrigation,
Blackjack, collision avoidance, and lunar lander.
