  While variance reduction methods have shown great success in solving large
scale optimization problems, many of them suffer from accumulated errors and,
therefore, should periodically require the full gradient computation. In this
paper, we present a single-loop algorithm named SLEDGE (Single-Loop mEthoD for
Gradient Estimator) for finite-sum nonconvex optimization, which does not
require periodic refresh of the gradient estimator but achieves nearly optimal
gradient complexity. Unlike existing methods, SLEDGE has the advantage of
versatility; (i) second-order optimality, (ii) exponential convergence in the
PL region, and (iii) smaller complexity under less heterogeneity of data.
  We build an efficient federated learning algorithm by exploiting these
favorable properties. We show the first and second-order optimality of the
output and also provide analysis under PL conditions. When the local budget is
sufficiently large and clients are less (Hessian-)~heterogeneous, the algorithm
requires fewer communication rounds then existing methods such as FedAvg,
SCAFFOLD, and Mime. The superiority of our method is verified in numerical
experiments.
