  Offline reinforcement learning (RL) aims to infer sequential decision
policies using only offline datasets. This is a particularly difficult setup,
especially when learning to achieve multiple different goals or outcomes under
a given scenario with only sparse rewards. For offline learning of
goal-conditioned policies via supervised learning, previous work has shown that
an advantage weighted log-likelihood loss guarantees monotonic policy
improvement. In this work we argue that, despite its benefits, this approach is
still insufficient to fully address the distribution shift and multi-modality
problems. The latter is particularly severe in long-horizon tasks where finding
a unique and optimal policy that goes from a state to the desired goal is
challenging as there may be multiple and potentially conflicting solutions. To
tackle these challenges, we propose a complementary advantage-based weighting
scheme that introduces an additional source of inductive bias: given a
value-based partitioning of the state space, the contribution of actions
expected to lead to target regions that are easier to reach, compared to the
final goal, is further increased. Empirically, we demonstrate that the proposed
approach, Dual-Advantage Weighted Offline Goal-conditioned RL (DAWOG),
outperforms several competing offline algorithms in commonly used benchmarks.
Analytically, we offer a guarantee that the learnt policy is never worse than
the underlying behaviour policy.
