  The core challenge of offline reinforcement learning (RL) is dealing with the
(potentially catastrophic) extrapolation error induced by the distribution
shift between the history dataset and the desired policy. A large portion of
prior work tackles this challenge by implicitly/explicitly regularizing the
learning policy towards the behavior policy, which is hard to estimate reliably
in practice. In this work, we propose to regularize towards the Q-function of
the behavior policy instead of the behavior policy itself, under the premise
that the Q-function can be estimated more reliably and easily by a SARSA-style
estimate and handles the extrapolation error more straightforwardly. We propose
two algorithms taking advantage of the estimated Q-function through
regularizations, and demonstrate they exhibit strong performance on the D4RL
benchmarks.
