  Offline constrained reinforcement learning (RL) aims to learn a policy that
maximizes the expected cumulative reward subject to constraints on expected
cumulative cost using an existing dataset. In this paper, we propose
Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained
RL with general function approximation. PDCA runs a primal-dual algorithm on
the Lagrangian function estimated by critics. The primal player employs a
no-regret policy optimization oracle to maximize the Lagrangian estimate and
the dual player acts greedily to minimize the Lagrangian estimate. We show that
PDCA can successfully find a near saddle point of the Lagrangian, which is
nearly optimal for the constrained RL problem. Unlike previous work that
requires concentrability and a strong Bellman completeness assumption, PDCA
only requires concentrability and realizability assumptions for
sample-efficient learning.
