  Stein variational gradient descent (SVGD) is a non-parametric inference
algorithm that evolves a set of particles to fit a given distribution of
interest. We analyze the non-asymptotic properties of SVGD, showing that there
exists a set of functions, which we call the Stein matching set, whose
expectations are exactly estimated by any set of particles that satisfies the
fixed point equation of SVGD. This set is the image of Stein operator applied
on the feature maps of the positive definite kernel used in SVGD. Our results
provide a theoretical framework for analyzing the properties of SVGD with
different kernels, shedding insight into optimal kernel choice. In particular,
we show that SVGD with linear kernels yields exact estimation of means and
variances on Gaussian distributions, while random Fourier features enable
probabilistic bounds for distributional approximation. Our results offer a
refreshing view of the classical inference problem as fitting Stein's identity
or solving the Stein equation, which may motivate more efficient algorithms.
