  While deep reinforcement learning has proven to be successful in solving
control tasks, the "black-box" nature of an agent has received increasing
concerns. We propose a prototype-based post-hoc policy explainer, ProtoX, that
explains a blackbox agent by prototyping the agent's behaviors into scenarios,
each represented by a prototypical state. When learning prototypes, ProtoX
considers both visual similarity and scenario similarity. The latter is unique
to the reinforcement learning context, since it explains why the same action is
taken in visually different states. To teach ProtoX about visual similarity, we
pre-train an encoder using contrastive learning via self-supervised learning to
recognize states as similar if they occur close together in time and receive
the same action from the black-box agent. We then add an isometry layer to
allow ProtoX to adapt scenario similarity to the downstream task. ProtoX is
trained via imitation learning using behavior cloning, and thus requires no
access to the environment or agent. In addition to explanation fidelity, we
design different prototype shaping terms in the objective function to encourage
better interpretability. We conduct various experiments to test ProtoX. Results
show that ProtoX achieved high fidelity to the original black-box agent while
providing meaningful and understandable explanations.
