  Reinforcement learning (RL) has made a lot of advances for solving a single
problem in a given environment; but learning policies that generalize to unseen
variations of a problem remains challenging. To improve sample efficiency for
learning on such instances of a problem domain, we present Self-Paced Context
Evaluation (SPaCE). Based on self-paced learning, \spc automatically generates
\task curricula online with little computational overhead. To this end, SPaCE
leverages information contained in state values during training to accelerate
and improve training performance as well as generalization capabilities to new
instances from the same problem domain. Nevertheless, SPaCE is independent of
the problem domain at hand and can be applied on top of any RL agent with
state-value function approximation. We demonstrate SPaCE's ability to speed up
learning of different value-based RL agents on two environments, showing better
generalization capabilities and up to 10x faster learning compared to naive
approaches such as round robin or SPDRL, as the closest state-of-the-art
approach.
