  The task of estimating the gradient of a function in the presence of noise is
central to several forms of reinforcement learning, including policy search
methods. We present two techniques for reducing gradient estimation errors in
the presence of observable input noise applied to the control signal. The first
method extends the idea of a reinforcement baseline by fitting a local linear
model to the function whose gradient is being estimated; we show how to find
the linear model that minimizes the variance of the gradient estimate, and how
to estimate the model from data. The second method improves this further by
discounting components of the gradient vector that have high variance. These
methods are applied to the problem of motor control learning, where actuator
noise has a significant influence on behavior. In particular, we apply the
techniques to learn locally optimal controllers for a dart-throwing task using
a simulated three-link arm; we demonstrate that proposed methods significantly
improve the reward function gradient estimate and, consequently, the learning
curve, over existing methods.
