  We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian
inference method that generalizes Stein Variational Gradient Descent (SVGD) to
Riemann manifold. The benefits are two-folds: (i) for inference tasks in
Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information
geometry, and (ii) for inference tasks on Riemann manifolds, RSVGD brings the
unique advantages of SVGD to the Riemannian world. To appropriately transfer to
Riemann manifolds, we conceive novel and non-trivial techniques for RSVGD,
which are required by the intrinsically different characteristics of general
Riemann manifolds from Euclidean spaces. We also discover Riemannian Stein's
Identity and Riemannian Kernelized Stein Discrepancy. Experimental results show
the advantages over SVGD of exploring distribution geometry and the advantages
of particle-efficiency, iteration-effectiveness and approximation flexibility
over other inference methods on Riemann manifolds.
