  Reinforcement learning (RL) is used in many domains, including autonomous
driving, robotics, stock trading, and video games. Unfortunately, the black box
nature of RL agents, combined with legal and ethical considerations, makes it
increasingly important that humans (including those are who not experts in RL)
understand the reasoning behind the actions taken by an RL agent, particularly
in safety-critical domains. To help address this challenge, we introduce
PolicyExplainer, a visual analytics interface which lets the user directly
query an autonomous agent. PolicyExplainer visualizes the states, policy, and
expected future rewards for an agent, and supports asking and answering
questions such as: Why take this action? Why not take this other action? When
is this action taken? PolicyExplainer is designed based upon a domain analysis
with RL researchers, and is evaluated via qualitative and quantitative
assessments on a trio of domains: taxi navigation, a stack bot domain, and drug
recommendation for HIV patients. We find that PolicyExplainer promotes trust
and understanding of agent decisions better than a state-of-the-art text-based
explanation approach. Interviews with domain practitioners provide further
validation for PolicyExplainer as applied to safety-critical domains. Our
results help demonstrate how visualization-based approaches can be leveraged to
decode the behavior of autonomous RL agents, particularly for RL non-experts.
