  Learning policies from fixed offline datasets is a key challenge to scale up
reinforcement learning (RL) algorithms towards practical applications. This is
often because off-policy RL algorithms suffer from distributional shift, due to
mismatch between dataset and the target policy, leading to high variance and
over-estimation of value functions. In this work, we propose variance
regularization for offline RL algorithms, using stationary distribution
corrections. We show that by using Fenchel duality, we can avoid double
sampling issues for computing the gradient of the variance regularizer. The
proposed algorithm for offline variance regularization (OVAR) can be used to
augment any existing offline policy optimization algorithms. We show that the
regularizer leads to a lower bound to the offline policy optimization
objective, which can help avoid over-estimation errors, and explains the
benefits of our approach across a range of continuous control domains when
compared to existing state-of-the-art algorithms.
