  While various multi-agent reinforcement learning methods have been proposed
in cooperative settings, few works investigate how self-interested learning
agents achieve mutual coordination in decentralized general-sum games and
generalize pre-trained policies to non-cooperative opponents during execution.
In this paper, we present Generalizable Risk-Sensitive Policy (GRSP). GRSP
learns the distributions over agent's return and estimate a dynamic
risk-seeking bonus to discover risky coordination strategies. Furthermore, to
avoid overfitting to training opponents, GRSP learns an auxiliary opponent
modeling task to infer opponents' types and dynamically alter corresponding
strategies during execution. Empirically, agents trained via GRSP can achieve
mutual coordination during training stably and avoid being exploited by
non-cooperative opponents during execution. To the best of our knowledge, it is
the first method to learn coordination strategies between agents both in
iterated prisoner's dilemma (IPD) and iterated stag hunt (ISH) without shaping
opponents or rewards, and firstly consider generalization during execution.
Furthermore, we show that GRSP can be scaled to high-dimensional settings.
