  The Credit Assignment Problem (CAP) refers to the longstanding challenge of
Reinforcement Learning (RL) agents to associate actions with their long-term
consequences. Solving the CAP is a crucial step towards the successful
deployment of RL in the real world since most decision problems provide
feedback that is noisy, delayed, and with little or no information about the
causes. These conditions make it hard to distinguish serendipitous outcomes
from those caused by informed decision-making. However, the mathematical nature
of credit and the CAP remains poorly understood and defined. In this survey, we
review the state of the art of Temporal Credit Assignment (CA) in deep RL. We
propose a unifying formalism for credit that enables equitable comparisons of
state-of-the-art algorithms and improves our understanding of the trade-offs
between the various methods. We cast the CAP as the problem of learning the
influence of an action over an outcome from a finite amount of experience. We
discuss the challenges posed by delayed effects, transpositions, and a lack of
action influence, and analyse how existing methods aim to address them.
Finally, we survey the protocols to evaluate a credit assignment method and
suggest ways to diagnose the sources of struggle for different methods.
Overall, this survey provides an overview of the field for new-entry
practitioners and researchers, it offers a coherent perspective for scholars
looking to expedite the starting stages of a new study on the CAP, and it
suggests potential directions for future research.
