  Policies trained via reinforcement learning (RL) are often very complex even
for simple tasks. In an episode with $n$ time steps, a policy will make $n$
decisions on actions to take, many of which may appear non-intuitive to the
observer. Moreover, it is not clear which of these decisions directly
contribute towards achieving the reward and how significant is their
contribution. Given a trained policy, we propose a black-box method based on
counterfactual reasoning that estimates the causal effect that these decisions
have on reward attainment and ranks the decisions according to this estimate.
In this preliminary work, we compare our measure against an alternative,
non-causal, ranking procedure, highlight the benefits of causality-based policy
ranking, and discuss potential future work integrating causal algorithms into
the interpretation of RL agent policies.
