  We introduce Mix&Match (M&M) - a training framework designed to facilitate
rapid and effective learning in RL agents, especially those that would be too
slow or too challenging to train otherwise. The key innovation is a procedure
that allows us to automatically form a curriculum over agents. Through such a
curriculum we can progressively train more complex agents by, effectively,
bootstrapping from solutions found by simpler agents. In contradistinction to
typical curriculum learning approaches, we do not gradually modify the tasks or
environments presented, but instead use a process to gradually alter how the
policy is represented internally. We show the broad applicability of our method
by demonstrating significant performance gains in three different experimental
setups: (1) We train an agent able to control more than 700 actions in a
challenging 3D first-person task; using our method to progress through an
action-space curriculum we achieve both faster training and better final
performance than one obtains using traditional methods. (2) We further show
that M&M can be used successfully to progress through a curriculum of
architectural variants defining an agents internal state. (3) Finally, we
illustrate how a variant of our method can be used to improve agent performance
in a multitask setting.
