  As one of the prevalent methods to achieve automation systems, Imitation
Learning (IL) presents a promising performance in a wide range of domains.
However, despite the considerable improvement in policy performance, the
corresponding research on the explainability of IL models is still limited.
Inspired by the recent approaches in explainable artificial intelligence
methods, we proposed a model-agnostic explaining framework for IL models called
R2RISE. R2RISE aims to explain the overall policy performance with respect to
the frames in demonstrations. It iteratively retrains the black-box IL model
from the randomized masked demonstrations and uses the conventional evaluation
outcome environment returns as the coefficient to build an importance map. We
also conducted experiments to investigate three major questions concerning
frames' importance equality, the effectiveness of the importance map, and
connections between importance maps from different IL models. The result shows
that R2RISE successfully distinguishes important frames from the
demonstrations.
