  Emergent cooperation among self-interested individuals is a widespread
phenomenon in the natural world, but remains elusive in interactions between
artificially intelligent agents. Instead, na\"ive reinforcement learning
algorithms typically converge to Pareto-dominated outcomes in even the simplest
of social dilemmas. An emerging class of opponent-shaping methods have
demonstrated the ability to reach prosocial outcomes by influencing the
learning of other agents. However, they rely on higher-order derivatives
through the predicted learning step of other agents or learning meta-game
dynamics, which in turn rely on stringent assumptions over opponent learning
rules or exponential sample complexity, respectively. To provide a learning
rule-agnostic and sample-efficient alternative, we introduce Reciprocators,
reinforcement learning agents which are intrinsically motivated to reciprocate
the influence of an opponent's actions on their returns. This approach
effectively seeks to modify other agents' $Q$-values by increasing their return
following beneficial actions (with respect to the Reciprocator) and decreasing
it after detrimental actions, guiding them towards mutually beneficial actions
without attempting to directly shape policy updates. We show that Reciprocators
can be used to promote cooperation in a variety of temporally extended social
dilemmas during simultaneous learning.
