  In order for agents in multi-agent systems (MAS) to be safe, they need to
take into account the risks posed by the actions of other agents. However, the
dominant paradigm in game theory (GT) assumes that agents are not affected by
risk from other agents and only strive to maximise their expected utility. For
example, in hybrid human-AI driving systems, it is necessary to limit large
deviations in reward resulting from car crashes. Although there are equilibrium
concepts in game theory that take into account risk aversion, they either
assume that agents are risk-neutral with respect to the uncertainty caused by
the actions of other agents, or they are not guaranteed to exist. We introduce
a new GT-based Risk-Averse Equilibrium (RAE) that always produces a solution
that minimises the potential variance in reward accounting for the strategy of
other agents. Theoretically and empirically, we show RAE shares many properties
with a Nash Equilibrium (NE), establishing convergence properties and
generalising to risk-dominant NE in certain cases. To tackle large-scale
problems, we extend RAE to the PSRO multi-agent reinforcement learning (MARL)
framework. We empirically demonstrate the minimum reward variance benefits of
RAE in matrix games with high-risk outcomes. Results on MARL experiments show
RAE generalises to risk-dominant NE in a trust dilemma game and that it reduces
instances of crashing by 7x in an autonomous driving setting versus the best
performing baseline.
