  Robot navigation with deep reinforcement learning (RL) achieves higher
performance and performs well under complex environment. Meanwhile, the
interpretation of the decision-making of deep RL models becomes a critical
problem for more safety and reliability of autonomous robots. In this paper, we
propose a visual explanation method based on an attention branch for deep RL
models. We connect attention branch with pre-trained deep RL model and the
attention branch is trained by using the selected action by the trained deep RL
model as a correct label in a supervised learning manner. Because the attention
branch is trained to output the same result as the deep RL model, the obtained
attention maps are corresponding to the agent action with higher
interpretability. Experimental results with robot navigation task show that the
proposed method can generate interpretable attention maps for a visual
explanation.
