  Reparameterization (RP) and likelihood ratio (LR) gradient estimators are
used throughout machine and reinforcement learning; however, they are usually
explained as simple mathematical tricks without providing any insight into
their nature. We use a first principles approach to explain LR and RP, and show
a connection between the two via the divergence theorem. The theory motivated
us to derive optimal importance sampling schemes to reduce LR gradient
variance. Our newly derived distributions have analytic probability densities
and can be directly sampled from. The improvement for Gaussian target
distributions was modest, but for other distributions such as a Beta
distribution, our method could lead to arbitrarily large improvements, and was
crucial to obtain competitive performance in evolution strategies experiments.
