  Estimating the score, i.e., the gradient of log density function, from a set
of samples generated by an unknown distribution is a fundamental task in
inference and learning of probabilistic models that involve flexible yet
intractable densities. Kernel estimators based on Stein's methods or score
matching have shown promise, however their theoretical properties and
relationships have not been fully-understood. We provide a unifying view of
these estimators under the framework of regularized nonparametric regression.
It allows us to analyse existing estimators and construct new ones with
desirable properties by choosing different hypothesis spaces and regularizers.
A unified convergence analysis is provided for such estimators. Finally, we
propose score estimators based on iterative regularization that enjoy
computational benefits from curl-free kernels and fast convergence.
