  The main challenge in developing effective reinforcement learning (RL)
pipelines is often the design and tuning the reward functions. Well-designed
shaping reward can lead to significantly faster learning. Naively formulated
rewards, however, can conflict with the desired behavior and result in
overfitting or even erratic performance if not properly tuned. In theory, the
broad class of potential based reward shaping (PBRS) can help guide the
learning process without affecting the optimal policy. Although several studies
have explored the use of potential based reward shaping to accelerate learning
convergence, most have been limited to grid-worlds and low-dimensional systems,
and RL in robotics has predominantly relied on standard forms of reward
shaping. In this paper, we benchmark standard forms of shaping with PBRS for a
humanoid robot. We find that in this high-dimensional system, PBRS has only
marginal benefits in convergence speed. However, the PBRS reward terms are
significantly more robust to scaling than typical reward shaping approaches,
and thus easier to tune.
