  The excellent performance of Transformer in supervised learning has led to
growing interest in its potential application to deep reinforcement learning
(DRL) to achieve high performance on a wide variety of problems. However, the
decision making of a DRL agent is a black box, which greatly hinders the
application of the agent to real-world problems. To address this problem, we
propose the Action Q-Transformer (AQT), which introduces a transformer
encoder-decoder structure to Q-learning based DRL methods. In AQT, the encoder
calculates the state value function and the decoder calculates the advantage
function to promote the acquisition of different attentions indicating the
agent's decision-making. The decoder in AQT utilizes action queries, which
represent the information of each action, as queries. This enables us to obtain
the attentions for the state value and for each action. By acquiring and
visualizing these attentions that detail the agent's decision-making, we
achieve a DRL model with high interpretability. In this paper, we show that
visualization of attention in Atari 2600 games enables detailed analysis of
agents' decision-making in various game tasks. Further, experimental results
demonstrate that our method can achieve higher performance than the baseline in
some games.
