  Offline reinforcement learning (RL) provides a promising approach to avoid
costly online interaction with the real environment. However, the performance
of offline RL highly depends on the quality of the datasets, which may cause
extrapolation error in the learning process. In many robotic applications, an
inaccurate simulator is often available. However, the data directly collected
from the inaccurate simulator cannot be directly used in offline RL due to the
well-known exploration-exploitation dilemma and the dynamic gap between
inaccurate simulation and the real environment. To address these issues, we
propose a novel approach to combine the offline dataset and the inaccurate
simulation data in a better manner. Specifically, we pre-train a generative
adversarial network (GAN) model to fit the state distribution of the offline
dataset. Given this, we collect data from the inaccurate simulator starting
from the distribution provided by the generator and reweight the simulated data
using the discriminator. Our experimental results in the D4RL benchmark and a
real-world manipulation task confirm that our method can benefit more from both
inaccurate simulator and limited offline datasets to achieve better performance
than the state-of-the-art methods.
