  In recent years, there has been significant progress in applying deep
reinforcement learning (RL) for solving challenging problems across a wide
variety of domains. Nevertheless, convergence of various methods has been shown
to suffer from inconsistencies, due to algorithmic instability and variance, as
well as stochasticity in the benchmark environments. Particularly, despite the
fact that the agent's performance may be improving on average, it may abruptly
deteriorate at late stages of training. In this work, we study methods for
enhancing the agent's learning process, by providing conservative updates with
respect to either the obtained history or a reference benchmark policy. Our
method, termed EVEREST, obtains high confidence improvements via confidence
bounds of a reference policy. Through extensive empirical analysis we
demonstrate the benefit of our approach in terms of both performance and
stabilization, with significant improvements in continuous control and Atari
benchmarks.
