  In offline reinforcement learning (RL), the performance of the learned policy
highly depends on the quality of offline datasets. However, in many cases, the
offline dataset contains very limited optimal trajectories, which poses a
challenge for offline RL algorithms as agents must acquire the ability to
transit to high-reward regions. To address this issue, we introduce
Diffusion-based Trajectory Stitching (DiffStitch), a novel diffusion-based data
augmentation pipeline that systematically generates stitching transitions
between trajectories. DiffStitch effectively connects low-reward trajectories
with high-reward trajectories, forming globally optimal trajectories to address
the challenges faced by offline RL algorithms. Empirical experiments conducted
on D4RL datasets demonstrate the effectiveness of DiffStitch across RL
methodologies. Notably, DiffStitch demonstrates substantial enhancements in the
performance of one-step methods (IQL), imitation learning methods (TD3+BC), and
trajectory optimization methods (DT).
