  We study the Wasserstein natural gradient in parametric statistical models
with continuous sample spaces. Our approach is to pull back the
$L^2$-Wasserstein metric tensor in the probability density space to a parameter
space, equipping the latter with a positive definite metric tensor, under which
it becomes a Riemannian manifold, named the Wasserstein statistical manifold.
In general, it is not a totally geodesic sub-manifold of the density space, and
therefore its geodesics will differ from the Wasserstein geodesics, except for
the well-known Gaussian distribution case, a fact which can also be validated
under our framework. We use the sub-manifold geometry to derive a gradient flow
and natural gradient descent method in the parameter space. When parametrized
densities lie in $\bR$, the induced metric tensor establishes an explicit
formula. In optimization problems, we observe that the natural gradient descent
outperforms the standard gradient descent when the Wasserstein distance is the
objective function. In such a case, we prove that the resulting algorithm
behaves similarly to the Newton method in the asymptotic regime. The proof
calculates the exact Hessian formula for the Wasserstein distance, which
further motivates another preconditioner for the optimization process. To the
end, we present examples to illustrate the effectiveness of the natural
gradient in several parametric statistical models, including the Gaussian
measure, Gaussian mixture, Gamma distribution, and Laplace distribution.
