  In multi-agent reinforcement learning (MARL), ensuring robustness against
unpredictable or worst-case actions by allies is crucial for real-world
deployment. Existing robust MARL methods either approximate or enumerate all
possible threat scenarios against worst-case adversaries, leading to
computational intensity and reduced robustness. In contrast, human learning
efficiently acquires robust behaviors in daily life without preparing for every
possible threat. Inspired by this, we frame robust MARL as an inference
problem, with worst-case robustness implicitly optimized under all threat
scenarios via off-policy evaluation. Within this framework, we demonstrate that
Mutual Information Regularization as Robust Regularization (MIR3) during
routine training is guaranteed to maximize a lower bound on robustness, without
the need for adversaries. Further insights show that MIR3 acts as an
information bottleneck, preventing agents from over-reacting to others and
aligning policies with robust action priors. In the presence of worst-case
adversaries, our MIR3 significantly surpasses baseline methods in robustness
and training efficiency while maintaining cooperative performance in StarCraft
II and robot swarm control. When deploying the robot swarm control algorithm in
the real world, our method also outperforms the best baseline by 14.29%.
