  We introduce a new unsupervised pre-training method for reinforcement
learning called APT, which stands for Active Pre-Training. APT learns behaviors
and representations by actively searching for novel states in reward-free
environments. The key novel idea is to explore the environment by maximizing a
non-parametric entropy computed in an abstract representation space, which
avoids challenging density modeling and consequently allows our approach to
scale much better in environments that have high-dimensional observations
(e.g., image observations). We empirically evaluate APT by exposing
task-specific reward after a long unsupervised pre-training phase. In Atari
games, APT achieves human-level performance on 12 games and obtains highly
competitive performance compared to canonical fully supervised RL algorithms.
On DMControl suite, APT beats all baselines in terms of asymptotic performance
and data efficiency and dramatically improves performance on tasks that are
extremely difficult to train from scratch.
