  Stochastic variance reduction algorithms have recently become popular for
minimizing the average of a large, but finite, number of loss functions. In
this paper, we propose a novel Riemannian extension of the Euclidean stochastic
variance reduced gradient algorithm (R-SVRG) to a compact manifold search
space. To this end, we show the developments on the Grassmann manifold. The key
challenges of averaging, addition, and subtraction of multiple gradients are
addressed with notions like logarithm mapping and parallel translation of
vectors on the Grassmann manifold. We present a global convergence analysis of
the proposed algorithm with decay step-sizes and a local convergence rate
analysis under fixed step-size with some natural assumptions. The proposed
algorithm is applied on a number of problems on the Grassmann manifold like
principal components analysis, low-rank matrix completion, and the Karcher mean
computation. In all these cases, the proposed algorithm outperforms the
standard Riemannian stochastic gradient descent algorithm.
