  Reinforcement learning (RL) methods usually treat reward functions as black
boxes. As such, these methods must extensively interact with the environment in
order to discover rewards and optimal policies. In most RL applications,
however, users have to program the reward function and, hence, there is the
opportunity to make the reward function visible -- to show the reward
function's code to the RL agent so it can exploit the function's internal
structure to learn optimal policies in a more sample efficient manner. In this
paper, we show how to accomplish this idea in two steps. First, we propose
reward machines, a type of finite state machine that supports the specification
of reward functions while exposing reward function structure. We then describe
different methodologies to exploit this structure to support learning,
including automated reward shaping, task decomposition, and counterfactual
reasoning with off-policy learning. Experiments on tabular and continuous
domains, across different tasks and RL agents, show the benefits of exploiting
reward structure with respect to sample efficiency and the quality of resultant
policies. Finally, by virtue of being a form of finite state machine, reward
machines have the expressive power of a regular language and as such support
loops, sequences and conditionals, as well as the expression of temporally
extended properties typical of linear temporal logic and non-Markovian reward
specification.
