  Deep Policy Gradient (PG) algorithms employ value networks to drive the
learning of parameterized policies and reduce the variance of the gradient
estimates. However, value function approximation gets stuck in local optima and
struggles to fit the actual return, limiting the variance reduction efficacy
and leading policies to sub-optimal performance. This paper focuses on
improving value approximation and analyzing the effects on Deep PG primitives
such as value prediction, variance reduction, and correlation of gradient
estimates with the true gradient. To this end, we introduce a Value Function
Search that employs a population of perturbed value networks to search for a
better approximation. Our framework does not require additional environment
interactions, gradient computations, or ensembles, providing a computationally
inexpensive approach to enhance the supervised learning task on which value
networks train. Crucially, we show that improving Deep PG primitives results in
improved sample efficiency and policies with higher returns using common
continuous control benchmark domains.
