  Offline multi-agent reinforcement learning (MARL) aims to learn effective
multi-agent policies from pre-collected datasets, which is an important step
toward the deployment of multi-agent systems in real-world applications.
However, in practice, each individual behavior policy that generates
multi-agent joint trajectories usually has a different level of how well it
performs. e.g., an agent is a random policy while other agents are medium
policies. In the cooperative game with global reward, one agent learned by
existing offline MARL often inherits this random policy, jeopardizing the
performance of the entire team. In this paper, we investigate offline MARL with
explicit consideration on the diversity of agent-wise trajectories and propose
a novel framework called Shared Individual Trajectories (SIT) to address this
problem. Specifically, an attention-based reward decomposition network assigns
the credit to each agent through a differentiable key-value memory mechanism in
an offline manner. These decomposed credits are then used to reconstruct the
joint offline datasets into prioritized experience replay with individual
trajectories, thereafter agents can share their good trajectories and
conservatively train their policies with a graph attention network (GAT) based
critic. We evaluate our method in both discrete control (i.e., StarCraft II and
multi-agent particle environment) and continuous control (i.e, multi-agent
mujoco). The results indicate that our method achieves significantly better
results in complex and mixed offline multi-agent datasets, especially when the
difference of data quality between individual trajectories is large.
