  Reinforcement learning (RL) is a powerful technique for training intelligent
agents, but understanding why these agents make specific decisions can be quite
challenging. This lack of transparency in RL models has been a long-standing
problem, making it difficult for users to grasp the reasons behind an agent's
behaviour. Various approaches have been explored to address this problem, with
one promising avenue being reward decomposition (RD). RD is appealing as it
sidesteps some of the concerns associated with other methods that attempt to
rationalize an agent's behaviour in a post-hoc manner. RD works by exposing
various facets of the rewards that contribute to the agent's objectives during
training. However, RD alone has limitations as it primarily offers insights
based on sub-rewards and does not delve into the intricate cause-and-effect
relationships that occur within an RL agent's neural model. In this paper, we
present an extension of RD that goes beyond sub-rewards to provide more
informative explanations. Our approach is centred on a causal learning
framework that leverages information-theoretic measures for explanation
objectives that encourage three crucial properties of causal factors: causal
sufficiency, sparseness, and orthogonality. These properties help us distill
the cause-and-effect relationships between the agent's states and actions or
rewards, allowing for a deeper understanding of its decision-making processes.
Our framework is designed to generate local explanations and can be applied to
a wide range of RL tasks with multiple reward channels. Through a series of
experiments, we demonstrate that our approach offers more meaningful and
insightful explanations for the agent's action selections.
