  The challenge of developing powerful and general Reinforcement Learning (RL)
agents has received increasing attention in recent years. Much of this effort
has focused on the single-agent setting, in which an agent maximizes a
predefined extrinsic reward function. However, a long-term question inevitably
arises: how will such independent agents cooperate when they are continually
learning and acting in a shared multi-agent environment? Observing that humans
often provide incentives to influence others' behavior, we propose to equip
each RL agent in a multi-agent environment with the ability to give rewards
directly to other agents, using a learned incentive function. Each agent learns
its own incentive function by explicitly accounting for its impact on the
learning of recipients and, through them, the impact on its own extrinsic
objective. We demonstrate in experiments that such agents significantly
outperform standard RL and opponent-shaping agents in challenging general-sum
Markov games, often by finding a near-optimal division of labor. Our work
points toward more opportunities and challenges along the path to ensure the
common good in a multi-agent future.
