  Reinforcement Learning (RL) systems can be complex and non-interpretable,
making it challenging for non-AI experts to understand or intervene in their
decisions. This is due in part to the sequential nature of RL in which actions
are chosen because of future rewards. However, RL agents discard the
qualitative features of their training, making it difficult to recover
user-understandable information for "why" an action is chosen. We propose a
technique, Experiential Explanations, to generate counterfactual explanations
by training influence predictors along with the RL policy. Influence predictors
are models that learn how sources of reward affect the agent in different
states, thus restoring information about how the policy reflects the
environment. A human evaluation study revealed that participants presented with
experiential explanations were better able to correctly guess what an agent
would do than those presented with other standard types of explanation.
Participants also found that experiential explanations are more understandable,
satisfying, complete, useful, and accurate. The qualitative analysis provides
insights into the factors of experiential explanations that are most useful.
