  Learning effective policies for sparse objectives is a key challenge in Deep
Reinforcement Learning (RL). A common approach is to design task-related dense
rewards to improve task learnability. While such rewards are easily
interpreted, they rely on heuristics and domain expertise. Alternate approaches
that train neural networks to discover dense surrogate rewards avoid
heuristics, but are high-dimensional, black-box solutions offering little
interpretability. In this paper, we present a method that discovers dense
rewards in the form of low-dimensional symbolic trees - thus making them more
tractable for analysis. The trees use simple functional operators to map an
agent's observations to a scalar reward, which then supervises the policy
gradient learning of a neural network policy. We test our method on continuous
action spaces in Mujoco and discrete action spaces in Atari and Pygame
environments. We show that the discovered dense rewards are an effective signal
for an RL policy to solve the benchmark tasks. Notably, we significantly
outperform a widely used, contemporary neural-network based reward-discovery
algorithm in all environments considered.
