  Recently, the use of transformers in offline reinforcement learning has
become a rapidly developing area. This is due to their ability to treat the
agent's trajectory in the environment as a sequence, thereby reducing the
policy learning problem to sequence modeling. In environments where the agent's
decisions depend on past events (POMDPs), capturing both the event itself and
the decision point in the context of the model is essential. However, the
quadratic complexity of the attention mechanism limits the potential for
context expansion. One solution to this problem is to enhance transformers with
memory mechanisms. This paper proposes a Recurrent Action Transformer with
Memory (RATE), a novel model architecture incorporating a recurrent memory
mechanism designed to regulate information retention. To evaluate our model, we
conducted extensive experiments on memory-intensive environments
(ViZDoom-Two-Colors, T-Maze, Memory Maze, Minigrid.Memory), classic Atari games
and MuJoCo control environments. The results show that using memory can
significantly improve performance in memory-intensive environments while
maintaining or improving results in classic environments. We hope our findings
will stimulate research on memory mechanisms for transformers applicable to
offline reinforcement learning.
