  Quality-Diversity (QD) is a concept from Neuroevolution with some intriguing
applications to Reinforcement Learning. It facilitates learning a population of
agents where each member is optimized to simultaneously accumulate high
task-returns and exhibit behavioral diversity compared to other members. In
this paper, we build on a recent kernel-based method for training a QD policy
ensemble with Stein variational gradient descent. With kernels based on
$f$-divergence between the stationary distributions of policies, we convert the
problem to that of efficient estimation of the ratio of these stationary
distributions. We then study various distribution ratio estimators used
previously for off-policy evaluation and imitation and re-purpose them to
compute the gradients for policies in an ensemble such that the resultant
population is diverse and of high-quality.
