  Offline reinforcement learning (RL) learns policies entirely from static
datasets, thereby avoiding the challenges associated with online data
collection. Practical applications of offline RL will inevitably require
learning from datasets where the variability of demonstrated behaviors changes
non-uniformly across the state space. For example, at a red light, nearly all
human drivers behave similarly by stopping, but when merging onto a highway,
some drivers merge quickly, efficiently, and safely, while many hesitate or
merge dangerously. Both theoretically and empirically, we show that typical
offline RL methods, which are based on distribution constraints fail to learn
from data with such non-uniform variability, due to the requirement to stay
close to the behavior policy to the same extent across the state space.
Ideally, the learned policy should be free to choose per state how closely to
follow the behavior policy to maximize long-term return, as long as the learned
policy stays within the support of the behavior policy. To instantiate this
principle, we reweight the data distribution in conservative Q-learning (CQL)
to obtain an approximate support constraint formulation. The reweighted
distribution is a mixture of the current policy and an additional policy
trained to mine poor actions that are likely under the behavior policy. Our
method, CQL (ReDS), is simple, theoretically motivated, and improves
performance across a wide range of offline RL problems in Atari games,
navigation, and pixel-based manipulation.
