  Understanding the agent's learning process, particularly the factors that
contribute to its success or failure post-training, is crucial for
comprehending the rationale behind the agent's decision-making process. Prior
methods clarify the learning process by creating a structural causal model
(SCM) or visually representing the distribution of value functions.
Nevertheless, these approaches have constraints as they exclusively function in
2D-environments or with uncomplicated transition dynamics. Understanding the
agent's learning process in complicated environments or tasks is more
challenging. In this paper, we propose REVEAL-IT, a novel framework for
explaining the learning process of an agent in complex environments. Initially,
we visualize the policy structure and the agent's learning process for various
training tasks. By visualizing these findings, we can understand how much a
particular training task or stage affects the agent's performance in test.
Then, a GNN-based explainer learns to highlight the most important section of
the policy, providing a more clear and robust explanation of the agent's
learning process. The experiments demonstrate that explanations derived from
this framework can effectively help in the optimization of the training tasks,
resulting in improved learning efficiency and final performance.
