  The goal of reinforcement learning (RL) is to find a policy that maximizes
the expected cumulative return. It has been shown that this objective can be
represented as an optimization problem of state-action visitation distribution
under linear constraints. The dual problem of this formulation, which we refer
to as dual RL, is unconstrained and easier to optimize. In this work, we first
cast several state-of-the-art offline RL and offline imitation learning (IL)
algorithms as instances of dual RL approaches with shared structures. Such
unification allows us to identify the root cause of the shortcomings of prior
methods. For offline IL, our analysis shows that prior methods are based on a
restrictive coverage assumption that greatly limits their performance in
practice. To fix this limitation, we propose a new discriminator-free method
ReCOIL that learns to imitate from arbitrary off-policy data to obtain
near-expert performance. For offline RL, our analysis frames a recent offline
RL method XQL in the dual framework, and we further propose a new method f-DVL
that provides alternative choices to the Gumbel regression loss that fixes the
known training instability issue of XQL. The performance improvements by both
of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an
extensive suite of simulated robot locomotion and manipulation tasks. Project
code and details can be found at this https://hari-sikchi.github.io/dual-rl.
