  This article provides the first survey of computational models of emotion in
reinforcement learning (RL) agents. The survey focuses on agent/robot emotions,
and mostly ignores human user emotions. Emotions are recognized as functional
in decision-making by influencing motivation and action selection. Therefore,
computational emotion models are usually grounded in the agent's decision
making architecture, of which RL is an important subclass. Studying emotions in
RL-based agents is useful for three research fields. For machine learning (ML)
researchers, emotion models may improve learning efficiency. For the
interactive ML and human-robot interaction (HRI) community, emotions can
communicate state and enhance user investment. Lastly, it allows affective
modelling (AM) researchers to investigate their emotion theories in a
successful AI agent class. This survey provides background on emotion theory
and RL. It systematically addresses 1) from what underlying dimensions (e.g.,
homeostasis, appraisal) emotions can be derived and how these can be modelled
in RL-agents, 2) what types of emotions have been derived from these
dimensions, and 3) how these emotions may either influence the learning
efficiency of the agent or be useful as social signals. We also systematically
compare evaluation criteria, and draw connections to important RL sub-domains
like (intrinsic) motivation and model-based RL. In short, this survey provides
both a practical overview for engineers wanting to implement emotions in their
RL agents, and identifies challenges and directions for future emotion-RL
research.
