  This paper investigates the use of prior computation to estimate the value
function to improve sample efficiency in on-policy policy gradient methods in
reinforcement learning. Our approach is to estimate the value function from
prior computations, such as from the Q-network learned in DQN or the value
function trained for different but related environments. In particular, we
learn a new value function for the target task while combining it with a value
estimate from the prior computation. Finally, the resulting value function is
used as a baseline in the policy gradient method. This use of a baseline has
the theoretical property of reducing variance in gradient computation and thus
improving sample efficiency. The experiments show the successful use of prior
value estimates in various settings and improved sample efficiency in several
tasks.
