  In high-dimensional state spaces, the usefulness of Reinforcement Learning
(RL) is limited by the problem of exploration. This issue has been addressed
using potential-based reward shaping (PB-RS) previously. In the present work,
we introduce Final-Volume-Preserving Reward Shaping (FV-RS). FV-RS relaxes the
strict optimality guarantees of PB-RS to a guarantee of preserved long-term
behavior. Being less restrictive, FV-RS allows for reward shaping functions
that are even better suited for improving the sample efficiency of RL
algorithms. In particular, we consider settings in which the agent has access
to an approximate plan. Here, we use examples of simulated robotic manipulation
tasks to demonstrate that plan-based FV-RS can indeed significantly improve the
sample efficiency of RL over plan-based PB-RS.
