  Offline reinforcement learning endeavors to leverage offline datasets to
craft effective agent policy without online interaction, which imposes proper
conservative constraints with the support of behavior policies to tackle the
out-of-distribution problem. However, existing works often suffer from the
constraint conflict issue when offline datasets are collected from multiple
behavior policies, i.e., different behavior policies may exhibit inconsistent
actions with distinct returns across the state space. To remedy this issue,
recent advantage-weighted methods prioritize samples with high advantage values
for agent training while inevitably ignoring the diversity of behavior policy.
In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO)
method to explicitly construct advantage-aware policy constraints for offline
learning under mixed-quality datasets. Specifically, A2PO employs a conditional
variational auto-encoder to disentangle the action distributions of intertwined
behavior policies by modeling the advantage values of all training data as
conditional variables. Then the agent can follow such disentangled action
distribution constraints to optimize the advantage-aware policy towards high
advantage values. Extensive experiments conducted on both the single-quality
and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields
results superior to the counterparts. Our code will be made publicly available.
