  Reward shaping (RS) is a powerful method in reinforcement learning (RL) for
overcoming the problem of sparse or uninformative rewards. However, RS
typically relies on manually engineered shaping-reward functions whose
construction is time-consuming and error-prone. It also requires domain
knowledge which runs contrary to the goal of autonomous learning. We introduce
Reinforcement Learning Optimising Shaping Algorithm (ROSA), an automated reward
shaping framework in which the shaping-reward function is constructed in a
Markov game between two agents. A reward-shaping agent (Shaper) uses switching
controls to determine which states to add shaping rewards for more efficient
learning while the other agent (Controller) learns the optimal policy for the
task using these shaped rewards. We prove that ROSA, which adopts existing RL
algorithms, learns to construct a shaping-reward function that is beneficial to
the task thus ensuring efficient convergence to high performance policies. We
demonstrate ROSA's properties in three didactic experiments and show its
superior performance against state-of-the-art RS algorithms in challenging
sparse reward environments.
