  We introduce Autoverse, an evolvable, domain-specific language for
single-player 2D grid-based games, and demonstrate its use as a scalable
training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses
cellular-automaton-like rewrite rules to describe game mechanics, allowing it
to express various game environments (e.g. mazes, dungeons, sokoban puzzles)
that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite
rule can be expressed as a series of simple convolutions, allowing for
environments to be parallelized on the GPU, thereby drastically accelerating RL
training. Using Autoverse, we propose jump-starting open-ended learning by
imitation learning from search. In such an approach, we first evolve Autoverse
environments (their rules and initial map topology) to maximize the number of
iterations required by greedy tree search to discover a new best solution,
producing a curriculum of increasingly complex environments and playtraces. We
then distill these expert playtraces into a neural-network-based policy using
imitation learning. Finally, we use the learned policy as a starting point for
open-ended RL, where new training environments are continually evolved to
maximize the RL player agent's value function error (a proxy for its regret, or
the learnability of generated environments), finding that this approach
improves the performance and generality of resultant player agents.
