  In many real-world multi-agent cooperative tasks, due to high cost and risk,
agents cannot continuously interact with the environment and collect
experiences during learning, but have to learn from offline datasets. However,
the transition dynamics in the dataset of each agent can be much different from
the ones induced by the learned policies of other agents in execution, creating
large errors in value estimates. Consequently, agents learn uncoordinated
low-performing policies. In this paper, we propose a framework for offline
decentralized multi-agent reinforcement learning, which exploits value
deviation and transition normalization to deliberately modify the transition
probabilities. Value deviation optimistically increases the transition
probabilities of high-value next states, and transition normalization
normalizes the transition probabilities of next states. They together enable
agents to learn high-performing and coordinated policies. Theoretically, we
prove the convergence of Q-learning under the altered non-stationary transition
dynamics. Empirically, we show that the framework can be easily built on many
existing offline reinforcement learning algorithms and achieve substantial
improvement in a variety of multi-agent tasks.
