  In offline multi-agent reinforcement learning (MARL), agents estimate
policies from a given dataset. We study reward-poisoning attacks in this
setting where an exogenous attacker modifies the rewards in the dataset before
the agents see the dataset. The attacker wants to guide each agent into a
nefarious target policy while minimizing the $L^p$ norm of the reward
modification. Unlike attacks on single-agent RL, we show that the attacker can
install the target policy as a Markov Perfect Dominant Strategy Equilibrium
(MPDSE), which rational agents are guaranteed to follow. This attack can be
significantly cheaper than separate single-agent attacks. We show that the
attack works on various MARL agents including uncertainty-aware learners, and
we exhibit linear programs to efficiently solve the attack problem. We also
study the relationship between the structure of the datasets and the minimal
attack cost. Our work paves the way for studying defense in offline MARL.
