  Multi-agent cooperation is an important feature of the natural world. Many
tasks involve individual incentives that are misaligned with the common good,
yet a wide range of organisms from bacteria to insects and humans are able to
overcome their differences and collaborate. Therefore, the emergence of
cooperative behavior amongst self-interested individuals is an important
question for the fields of multi-agent reinforcement learning (MARL) and
evolutionary theory. Here, we study a particular class of multi-agent problems
called intertemporal social dilemmas (ISDs), where the conflict between the
individual and the group is particularly sharp. By combining MARL with
appropriately structured natural selection, we demonstrate that individual
inductive biases for cooperation can be learned in a model-free way. To achieve
this, we introduce an innovative modular architecture for deep reinforcement
learning agents which supports multi-level selection. We present results in two
challenging environments, and interpret these in the context of cultural and
ecological evolution.
