  We propose a new model-based offline RL framework, called Adversarial Models
for Offline Reinforcement Learning (ARMOR), which can robustly learn policies
to improve upon an arbitrary baseline policy regardless of data coverage. Based
on the concept of relative pessimism, ARMOR is designed to optimize for the
worst-case relative performance when facing uncertainty. In theory, we prove
that the learned policy of ARMOR never degrades the performance of the baseline
policy with any admissible hyperparameter, and can learn to compete with the
best policy within data coverage when the hyperparameter is well tuned, and the
baseline policy is supported by the data. Such a robust policy improvement
property makes ARMOR especially suitable for building real-world learning
systems, because in practice ensuring no performance degradation is imperative
before considering any benefit learning can bring.
