  Contracts are the economic framework which allows a principal to delegate a
task to an agent -- despite misaligned interests, and even without directly
observing the agent's actions. In many modern reinforcement learning settings,
self-interested agents learn to perform a multi-stage task delegated to them by
a principal. We explore the significant potential of utilizing contracts to
incentivize the agents. We model the delegated task as an MDP, and study a
stochastic game between the principal and agent where the principal learns what
contracts to use, and the agent learns an MDP policy in response. We present a
learning-based algorithm for optimizing the principal's contracts, which
provably converges to the subgame-perfect equilibrium of the principal-agent
game. A deep RL implementation allows us to apply our method to very large MDPs
with unknown transition dynamics. We extend our approach to multiple agents,
and demonstrate its relevance to resolving a canonical sequential social
dilemma with minimal intervention to agent rewards.
