  Decision Transformer (DT) is a recently proposed architecture for
Reinforcement Learning that frames the decision-making process as an
auto-regressive sequence modeling problem and uses a Transformer model to
predict the next action in a sequence of states, actions, and rewards. In this
paper, we analyze how crucial the Transformer model is in the complete DT
architecture on continuous control tasks. Namely, we replace the Transformer by
an LSTM model while keeping the other parts unchanged to obtain what we call a
Decision LSTM model. We compare it to DT on continuous control tasks, including
pendulum swing-up and stabilization, in simulation and on physical hardware.
Our experiments show that DT struggles with continuous control problems, such
as inverted pendulum and Furuta pendulum stabilization. On the other hand, the
proposed Decision LSTM is able to achieve expert-level performance on these
tasks, in addition to learning a swing-up controller on the real system. These
results suggest that the strength of the Decision Transformer for continuous
control tasks may lie in the overall sequential modeling architecture and not
in the Transformer per se.
