  Designing better deep networks and better reinforcement learning (RL)
algorithms are both important for deep RL. This work studies the former.
Specifically, the Perception and Decision-making Interleaving Transformer
(PDiT) network is proposed, which cascades two Transformers in a very natural
way: the perceiving one focuses on \emph{the environmental perception} by
processing the observation at the patch level, whereas the deciding one pays
attention to \emph{the decision-making} by conditioning on the history of the
desired returns, the perceiver's outputs, and the actions. Such a network
design is generally applicable to a lot of deep RL settings, e.g., both the
online and offline RL algorithms under environments with either image
observations, proprioception observations, or hybrid image-language
observations. Extensive experiments show that PDiT can not only achieve
superior performance than strong baselines in different settings but also
extract explainable feature representations. Our code is available at
\url{https://github.com/maohangyu/PDiT}.
