  Online interactions with the environment to collect data samples for training
a Reinforcement Learning (RL) agent is not always feasible due to economic and
safety concerns. The goal of Offline Reinforcement Learning is to address this
problem by learning effective policies using previously collected datasets.
Standard off-policy RL algorithms are prone to overestimations of the values of
out-of-distribution (less explored) actions and are hence unsuitable for
Offline RL. Behavior regularization, which constraints the learned policy
within the support set of the dataset, has been proposed to tackle the
limitations of standard off-policy algorithms. In this paper, we improve the
behavior regularized offline reinforcement learning and propose BRAC+. First,
we propose quantification of the out-of-distribution actions and conduct
comparisons between using Kullback-Leibler divergence versus using Maximum Mean
Discrepancy as the regularization protocol. We propose an analytical upper
bound on the KL divergence as the behavior regularizer to reduce variance
associated with sample based estimations. Second, we mathematically show that
the learned Q values can diverge even using behavior regularized policy update
under mild assumptions. This leads to large overestimations of the Q values and
performance deterioration of the learned policy. To mitigate this issue, we add
a gradient penalty term to the policy evaluation objective. By doing so, the Q
values are guaranteed to converge. On challenging offline RL benchmarks, BRAC+
outperforms the baseline behavior regularized approaches by 40%~87% and the
state-of-the-art approach by 6%.
