  Cross-device training is a crucial subfield of federated learning, where the
number of clients can reach into the billions. Standard approaches and local
methods are prone to issues such as client drift and insensitivity to data
similarities. We propose a novel algorithm (SPAM) for cross-device federated
learning with non-convex losses, which solves both issues. We provide sharp
analysis under second-order (Hessian) similarity, a condition satisfied by a
variety of machine learning problems in practice. Additionally, we extend our
results to the partial participation setting, where a cohort of selected
clients communicate with the server at each communication round. Our method is
the first in its kind, that does not require the smoothness of the objective
and provably benefits from clients having similar data.
