  Model-based offline reinforcement learning (RL) aims to find highly rewarding
policy, by leveraging a previously collected static dataset and a dynamics
model. While the dynamics model learned through reuse of the static dataset,
its generalization ability hopefully promotes policy learning if properly
utilized. To that end, several works propose to quantify the uncertainty of
predicted dynamics, and explicitly apply it to penalize reward. However, as the
dynamics and the reward are intrinsically different factors in context of MDP,
characterizing the impact of dynamics uncertainty through reward penalty may
incur unexpected tradeoff between model utilization and risk avoidance. In this
work, we instead maintain a belief distribution over dynamics, and
evaluate/optimize policy through biased sampling from the belief. The sampling
procedure, biased towards pessimism, is derived based on an alternating Markov
game formulation of offline RL. We formally show that the biased sampling
naturally induces an updated dynamics belief with policy-dependent reweighting
factor, termed Pessimism-Modulated Dynamics Belief. To improve policy, we
devise an iterative regularized policy optimization algorithm for the game,
with guarantee of monotonous improvement under certain condition. To make
practical, we further devise an offline RL algorithm to approximately find the
solution. Empirical results show that the proposed approach achieves
state-of-the-art performance on a wide range of benchmark tasks.
