  Constrained policy search (CPS) is a fundamental problem in offline
reinforcement learning, which is generally solved by advantage weighted
regression (AWR). However, previous methods may still encounter
out-of-distribution actions due to the limited expressivity of Gaussian-based
policies. On the other hand, directly applying the state-of-the-art models with
distribution expression capabilities (i.e., diffusion models) in the AWR
framework is intractable since AWR requires exact policy probability densities,
which is intractable in diffusion models. In this paper, we propose a novel
approach, $\textbf{Diffusion-based Constrained Policy Search}$ (dubbed
DiffCPS), which tackles the diffusion-based constrained policy search with the
primal-dual method. The theoretical analysis reveals that strong duality holds
for diffusion-based CPS problems, and upon introducing parameter approximation,
an approximated solution can be obtained after $\mathcal{O}(1/\epsilon)$ number
of dual iterations, where $\epsilon$ denotes the representation ability of the
parametrized policy. Extensive experimental results based on the D4RL benchmark
demonstrate the efficacy of our approach. We empirically show that DiffCPS
achieves better or at least competitive performance compared to traditional
AWR-based baselines as well as recent diffusion-based offline RL methods. The
code is now available at https://github.com/felix-thu/DiffCPS.
