  Multi-agent reinforcement learning (MARL) has shown recent success in
increasingly complex fixed-team zero-sum environments. However, the real world
is not zero-sum nor does it have fixed teams; humans face numerous social
dilemmas and must learn when to cooperate and when to compete. To successfully
deploy agents into the human world, it may be important that they be able to
understand and help in our conflicts. Unfortunately, selfish MARL agents
typically fail when faced with social dilemmas. In this work, we show evidence
of emergent direct reciprocity, indirect reciprocity and reputation, and team
formation when training agents with randomized uncertain social preferences
(RUSP), a novel environment augmentation that expands the distribution of
environments agents play in. RUSP is generic and scalable; it can be applied to
any multi-agent environment without changing the original underlying game
dynamics or objectives. In particular, we show that with RUSP these behaviors
can emerge and lead to higher social welfare equilibria in both classic
abstract social dilemmas like Iterated Prisoner's Dilemma as well in more
complex intertemporal environments.
