  Intelligent creatures can explore their environments and learn useful skills
without supervision. In this paper, we propose DIAYN ('Diversity is All You
Need'), a method for learning useful skills without a reward function. Our
proposed method learns skills by maximizing an information theoretic objective
using a maximum entropy policy. On a variety of simulated robotic tasks, we
show that this simple objective results in the unsupervised emergence of
diverse skills, such as walking and jumping. In a number of reinforcement
learning benchmark environments, our method is able to learn a skill that
solves the benchmark task despite never receiving the true task reward. We show
how pretrained skills can provide a good parameter initialization for
downstream tasks, and can be composed hierarchically to solve complex, sparse
reward tasks. Our results suggest that unsupervised discovery of skills can
serve as an effective pretraining mechanism for overcoming challenges of
exploration and data efficiency in reinforcement learning.
