  In offline model-based reinforcement learning (offline MBRL), we learn a
dynamic model from historically collected data, and subsequently utilize the
learned model and fixed datasets for policy learning, without further
interacting with the environment. Offline MBRL algorithms can improve the
efficiency and stability of policy learning over the model-free algorithms.
However, in most of the existing offline MBRL algorithms, the learning
objectives for the dynamic models and the policies are isolated from each
other. Such an objective mismatch may lead to inferior performance of the
learned agents. In this paper, we address this issue by developing an iterative
offline MBRL framework, where we maximize a lower bound of the true expected
return, by alternating between dynamic-model training and policy learning. With
the proposed unified model-policy learning framework, we achieve competitive
performance on a wide range of continuous-control offline reinforcement
learning datasets. Source code is publicly released.
