  Safe reinforcement learning (RL) trains a constraint satisfaction policy by
interacting with the environment. We aim to tackle a more challenging problem:
learning a safe policy from an offline dataset. We study the offline safe RL
problem from a novel multi-objective optimization perspective and propose the
$\epsilon$-reducible concept to characterize problem difficulties. The inherent
trade-offs between safety and task performance inspire us to propose the
constrained decision transformer (CDT) approach, which can dynamically adjust
the trade-offs during deployment. Extensive experiments show the advantages of
the proposed method in learning an adaptive, safe, robust, and high-reward
policy. CDT outperforms its variants and strong offline safe RL baselines by a
large margin with the same hyperparameters across all tasks, while keeping the
zero-shot adaptation capability to different constraint thresholds, making our
approach more suitable for real-world RL under constraints. The code is
available at https://github.com/liuzuxin/OSRL.
