  We study the variance of the REINFORCE policy gradient estimator in
environments with continuous state and action spaces, linear dynamics,
quadratic cost, and Gaussian noise. These simple environments allow us to
derive bounds on the estimator variance in terms of the environment and noise
parameters. We compare the predictions of our bounds to the empirical variance
in simulation experiments.
