  Offline reinforcement learning (RL) aims to find performant policies from
logged data without further environment interaction. Model-based algorithms,
which learn a model of the environment from the dataset and perform
conservative policy optimisation within that model, have emerged as a promising
approach to this problem. In this work, we present Robust Adversarial
Model-Based Offline RL (RAMBO), a novel approach to model-based offline RL. We
formulate the problem as a two-player zero sum game against an adversarial
environment model. The model is trained to minimise the value function while
still accurately predicting the transitions in the dataset, forcing the policy
to act conservatively in areas not covered by the dataset. To approximately
solve the two-player game, we alternate between optimising the policy and
adversarially optimising the model. The problem formulation that we address is
theoretically grounded, resulting in a probably approximately correct (PAC)
performance guarantee and a pessimistic value function which lower bounds the
value function in the true environment. We evaluate our approach on widely
studied offline RL benchmarks, and demonstrate that it outperforms existing
state-of-the-art baselines.
