  We study a probabilistic numerical method for the solution of both boundary
and initial value problems that returns a joint Gaussian process posterior over
the solution. Such methods have concrete value in the statistics on Riemannian
manifolds, where non-analytic ordinary differential equations are involved in
virtually all computations. The probabilistic formulation permits marginalising
the uncertainty of the numerical solution such that statistics are less
sensitive to inaccuracies. This leads to new Riemannian algorithms for mean
value computations and principal geodesic analysis. Marginalisation also means
results can be less precise than point estimates, enabling a noticeable
speed-up over the state of the art. Our approach is an argument for a wider
point that uncertainty caused by numerical calculations should be tracked
throughout the pipeline of machine learning algorithms.
