  We introduce Self-supervised Online Reward Shaping (SORS), which aims to
improve the sample efficiency of any RL algorithm in sparse-reward environments
by automatically densifying rewards. The proposed framework alternates between
classification-based reward inference and policy update steps -- the original
sparse reward provides a self-supervisory signal for reward inference by
ranking trajectories that the agent observes, while the policy update is
performed with the newly inferred, typically dense reward function. We
introduce theory that shows that, under certain conditions, this alteration of
the reward function will not change the optimal policy of the original MDP,
while potentially increasing learning speed significantly. Experimental results
on several sparse-reward environments demonstrate that, across multiple
domains, the proposed algorithm is not only significantly more sample efficient
than a standard RL baseline using sparse rewards, but, at times, also achieves
similar sample efficiency compared to when hand-designed dense reward functions
are used.
