  Guilt aversion induces experience of a utility loss in people if they believe
they have disappointed others, and this promotes cooperative behaviour in
human. In psychological game theory, guilt aversion necessitates modelling of
agents that have theory about what other agents think, also known as Theory of
Mind (ToM). We aim to build a new kind of affective reinforcement learning
agents, called Theory of Mind Agents with Guilt Aversion (ToMAGA), which are
equipped with an ability to think about the wellbeing of others instead of just
self-interest. To validate the agent design, we use a general-sum game known as
Stag Hunt as a test bed. As standard reinforcement learning agents could learn
suboptimal policies in social dilemmas like Stag Hunt, we propose to use
belief-based guilt aversion as a reward shaping mechanism. We show that our
belief-based guilt averse agents can efficiently learn cooperative behaviours
in Stag Hunt Games.
