  Deep policy networks enable robots to learn behaviors to solve various
real-world complex tasks in an end-to-end fashion. However, they lack
transparency to provide the reasons of actions. Thus, such a black-box model
often results in low reliability and disruptive actions during the deployment
of the robot in practice. To enhance its transparency, it is important to
explain robot behaviors by considering the extent to which each input feature
contributes to determining a given action. In this paper, we present an
explicit analysis of deep policy models through input attribution methods to
explain how and to what extent each input feature affects the decisions of the
robot policy models. To this end, we present two methods for applying input
attribution methods to robot policy networks: (1) we measure the importance
factor of each joint torque to reflect the influence of the motor torque on the
end-effector movement, and (2) we modify a relevance propagation method to
handle negative inputs and outputs in deep policy networks properly. To the
best of our knowledge, this is the first report to identify the dynamic changes
of input attributions of multi-modal sensor inputs in deep policy networks
online for robotic manipulation.
