  In this paper, we propose a novel sufficient decrease technique for
stochastic variance reduced gradient descent methods such as SVRG and SAGA. In
order to make sufficient decrease for stochastic optimization, we design a new
sufficient decrease criterion, which yields sufficient decrease versions of
stochastic variance reduction algorithms such as SVRG-SD and SAGA-SD as a
byproduct. We introduce a coefficient to scale current iterate and to satisfy
the sufficient decrease property, which takes the decisions to shrink, expand
or even move in the opposite direction, and then give two specific update rules
of the coefficient for Lasso and ridge regression. Moreover, we analyze the
convergence properties of our algorithms for strongly convex problems, which
show that our algorithms attain linear convergence rates. We also provide the
convergence guarantees of our algorithms for non-strongly convex problems. Our
experimental results further verify that our algorithms achieve significantly
better performance than their counterparts.
