  Decision Transformer (DT), which employs expressive sequence modeling
techniques to perform action generation, has emerged as a promising approach to
offline policy optimization. However, DT generates actions conditioned on a
desired future return, which is known to bear some weaknesses such as the
susceptibility to environmental stochasticity. To overcome DT's weaknesses, we
propose to empower DT with dynamic programming. Our method comprises three
steps. First, we employ in-sample value iteration to obtain approximated value
functions, which involves dynamic programming over the MDP structure. Second,
we evaluate action quality in context with estimated advantages. We introduce
two types of advantage estimators, IAE and GAE, which are suitable for
different tasks. Third, we train an Advantage-Conditioned Transformer (ACT) to
generate actions conditioned on the estimated advantages. Finally, during
testing, ACT generates actions conditioned on a desired advantage. Our
evaluation results validate that, by leveraging the power of dynamic
programming, ACT demonstrates effective trajectory stitching and robust action
generation in spite of the environmental stochasticity, outperforming baseline
methods across various benchmarks. Additionally, we conduct an in-depth
analysis of ACT's various design choices through ablation studies. Our code is
available at https://github.com/LAMDA-RL/ACT.
