  Recent progress in deep learning has relied on access to large and diverse
datasets. Such data-driven progress has been less evident in offline
reinforcement learning (RL), because offline RL data is usually collected to
optimize specific target tasks limiting the data's diversity. In this work, we
propose Exploratory data for Offline RL (ExORL), a data-centric approach to
offline RL. ExORL first generates data with unsupervised reward-free
exploration, then relabels this data with a downstream reward before training a
policy with offline RL. We find that exploratory data allows vanilla off-policy
RL algorithms, without any offline-specific modifications, to outperform or
match state-of-the-art offline RL algorithms on downstream tasks. Our findings
suggest that data generation is as important as algorithmic advances for
offline RL and hence requires careful consideration from the community. Code
and data can be found at https://github.com/denisyarats/exorl .
