  Offline multi-agent reinforcement learning (MARL) leverages static datasets
of experience to learn optimal multi-agent control. However, learning from
static data presents several unique challenges to overcome. In this paper, we
focus on coordination failure and investigate the role of joint actions in
multi-agent policy gradients with offline data, focusing on a common setting we
refer to as the 'Best Response Under Data' (BRUD) approach. By using two-player
polynomial games as an analytical tool, we demonstrate a simple yet overlooked
failure mode of BRUD-based algorithms, which can lead to catastrophic
coordination failure in the offline setting. Building on these insights, we
propose an approach to mitigate such failure, by prioritising samples from the
dataset based on joint-action similarity during policy learning and demonstrate
its effectiveness in detailed experiments. More generally, however, we argue
that prioritised dataset sampling is a promising area for innovation in offline
MARL that can be combined with other effective approaches such as critic and
policy regularisation. Importantly, our work shows how insights drawn from
simplified, tractable games can lead to useful, theoretically grounded insights
that transfer to more complex contexts. A core dimension of offering is an
interactive notebook, from which almost all of our results can be reproduced,
in a browser.
