  Self-play is an unsupervised training procedure which enables the
reinforcement learning agents to explore the environment without requiring any
external rewards. We augment the self-play setting by providing an external
memory where the agent can store experience from the previous tasks. This
enables the agent to come up with more diverse self-play tasks resulting in
faster exploration of the environment. The agent pretrained in the memory
augmented self-play setting easily outperforms the agent pretrained in
no-memory self-play setting.
