  In offline reinforcement learning (offline RL), one of the main challenges is
to deal with the distributional shift between the learning policy and the given
dataset. To address this problem, recent offline RL methods attempt to
introduce conservatism bias to encourage learning in high-confidence areas.
Model-free approaches directly encode such bias into policy or value function
learning using conservative regularizations or special network structures, but
their constrained policy search limits the generalization beyond the offline
dataset. Model-based approaches learn forward dynamics models with conservatism
quantifications and then generate imaginary trajectories to extend the offline
datasets. However, due to limited samples in offline datasets, conservatism
quantifications often suffer from overgeneralization in out-of-support regions.
The unreliable conservative measures will mislead forward model-based
imaginations to undesired areas, leading to overaggressive behaviors. To
encourage more conservatism, we propose a novel model-based offline RL
framework, called Reverse Offline Model-based Imagination (ROMI). We learn a
reverse dynamics model in conjunction with a novel reverse policy, which can
generate rollouts leading to the target goal states within the offline dataset.
These reverse imaginations provide informed data augmentation for model-free
policy learning and enable conservative generalization beyond the offline
dataset. ROMI can effectively combine with off-the-shelf model-free algorithms
to enable model-based generalization with proper conservatism. Empirical
results show that our method can generate more conservative behaviors and
achieve state-of-the-art performance on offline RL benchmark tasks.
