  Self-paced reinforcement learning (RL) aims to improve the data efficiency of
learning by automatically creating sequences, namely curricula, of probability
distributions over contexts. However, existing techniques for self-paced RL
fail in long-horizon planning tasks that involve temporally extended behaviors.
We hypothesize that taking advantage of prior knowledge about the underlying
task structure can improve the effectiveness of self-paced RL. We develop a
self-paced RL algorithm guided by reward machines, i.e., a type of finite-state
machine that encodes the underlying task structure. The algorithm integrates
reward machines in 1) the update of the policy and value functions obtained by
any RL algorithm of choice, and 2) the update of the automated curriculum that
generates context distributions. Our empirical results evidence that the
proposed algorithm achieves optimal behavior reliably even in cases in which
existing baselines cannot make any meaningful progress. It also decreases the
curriculum length and reduces the variance in the curriculum generation process
by up to one-fourth and four orders of magnitude, respectively.
