  In the future, artificial learning agents are likely to become increasingly
widespread in our society. They will interact with both other learning agents
and humans in a variety of complex settings including social dilemmas. We
consider the problem of how an external agent can promote cooperation between
artificial learners by distributing additional rewards and punishments based on
observing the learners' actions. We propose a rule for automatically learning
how to create right incentives by considering the players' anticipated
parameter updates. Using this learning rule leads to cooperation with high
social welfare in matrix games in which the agents would otherwise learn to
defect with high probability. We show that the resulting cooperative outcome is
stable in certain games even if the planning agent is turned off after a given
number of episodes, while other games require ongoing intervention to maintain
mutual cooperation. However, even in the latter case, the amount of necessary
additional incentives decreases over time.
