  Sampling a target probability distribution with an unknown normalization
constant is a fundamental challenge in computational science and engineering.
Recent work shows that algorithms derived by considering gradient flows in the
space of probability measures open up new avenues for algorithm development.
This paper makes three contributions to this sampling approach by scrutinizing
the design components of such gradient flows. Any instantiation of a gradient
flow for sampling needs an energy functional and a metric to determine the
flow, as well as numerical approximations of the flow to derive algorithms. Our
first contribution is to show that the Kullback-Leibler divergence, as an
energy functional, has the unique property (among all f-divergences) that
gradient flows resulting from it do not depend on the normalization constant of
the target distribution. Our second contribution is to study the choice of
metric from the perspective of invariance. The Fisher-Rao metric is known as
the unique choice (up to scaling) that is diffeomorphism invariant. As a
computationally tractable alternative, we introduce a relaxed, affine
invariance property for the metrics and gradient flows. In particular, we
construct various affine invariant Wasserstein and Stein gradient flows. Affine
invariant gradient flows are shown to behave more favorably than their
non-affine-invariant counterparts when sampling highly anisotropic
distributions, in theory and by using particle methods. Our third contribution
is to study, and develop efficient algorithms based on Gaussian approximations
of the gradient flows; this leads to an alternative to particle methods. We
establish connections between various Gaussian approximate gradient flows,
discuss their relation to gradient methods arising from parametric variational
inference, and study their convergence properties both theoretically and
numerically.
