  Generalization and adaptation of learned skills to novel situations is a core
requirement for intelligent autonomous robots. Although contextual
reinforcement learning provides a principled framework for learning and
generalization of behaviors across related tasks, it generally relies on
uninformed sampling of environments from an unknown, uncontrolled context
distribution, thus missing the benefits of structured, sequential learning. We
introduce a novel relative entropy reinforcement learning algorithm that gives
the agent the freedom to control the intermediate task distribution, allowing
for its gradual progression towards the target context distribution. Empirical
evaluation shows that the proposed curriculum learning scheme drastically
improves sample efficiency and enables learning in scenarios with both broad
and sharp target context distributions in which classical approaches perform
sub-optimally.
