  We propose discriminative reward co-training (DIRECT) as an extension to deep
reinforcement learning algorithms. Building upon the concept of self-imitation
learning (SIL), we introduce an imitation buffer to store beneficial
trajectories generated by the policy determined by their return. A
discriminator network is trained concurrently to the policy to distinguish
between trajectories generated by the current policy and beneficial
trajectories generated by previous policies. The discriminator's verdict is
used to construct a reward signal for optimizing the policy. By interpolating
prior experience, DIRECT is able to act as a surrogate, steering policy
optimization towards more valuable regions of the reward landscape thus
learning an optimal policy. Our results show that DIRECT outperforms
state-of-the-art algorithms in sparse- and shifting-reward environments being
able to provide a surrogate reward to the policy and direct the optimization
towards valuable areas.
