  In order to effectively interact with or supervise a robot, humans need to
have an accurate mental model of its capabilities and how it acts. Learned
neural network policies make that particularly challenging. We propose an
approach for helping end-users build a mental model of such policies. Our key
observation is that for most tasks, the essence of the policy is captured in a
few critical states: states in which it is very important to take a certain
action. Our user studies show that if the robot shows a human what its
understanding of the task's critical states is, then the human can make a more
informed decision about whether to deploy the policy, and if she does deploy
it, when she needs to take control from it at execution time.
