  In continuing tasks, average-reward reinforcement learning may be a more
appropriate problem formulation than the more common discounted reward
formulation. As usual, learning an optimal policy in this setting typically
requires a large amount of training experiences. Reward shaping is a common
approach for incorporating domain knowledge into reinforcement learning in
order to speed up convergence to an optimal policy. However, to the best of our
knowledge, the theoretical properties of reward shaping have thus far only been
established in the discounted setting. This paper presents the first reward
shaping framework for average-reward learning and proves that, under standard
assumptions, the optimal policy under the original reward function can be
recovered. In order to avoid the need for manual construction of the shaping
function, we introduce a method for utilizing domain knowledge expressed as a
temporal logic formula. The formula is automatically translated to a shaping
function that provides additional reward throughout the learning process. We
evaluate the proposed method on three continuing tasks. In all cases, shaping
speeds up the average-reward learning rate without any reduction in the
performance of the learned policy compared to relevant baselines.
