  Linear Temporal Logic (LTL) offers a precise means for constraining the
behavior of reinforcement learning agents. However, in many tasks, LTL is
insufficient for task specification; LTL-constrained policy optimization, where
the goal is to optimize a scalar reward under LTL constraints, is needed. Prior
methods for this constrained problem are restricted to finite state spaces. In
this work, we present Cycle Experience Replay (CyclER), a reward-shaping
approach to this problem that allows continuous state and action spaces and the
use of function approximations. CyclER guides a policy towards satisfaction by
encouraging partial behaviors compliant with the LTL constraint, using the
structure of the constraint. In doing so, it addresses the optimization
challenges stemming from the sparse nature of LTL satisfaction. We evaluate
CyclER in three continuous control domains. On these tasks, CyclER outperforms
existing reward-shaping methods at finding performant and LTL-satisfying
policies.
