  There are two paradigms in Federated Learning (FL): parallel FL (PFL), where
models are trained in a parallel manner across clients; and sequential FL
(SFL), where models are trained in a sequential manner across clients. In
contrast to that of PFL, the convergence theory of SFL on heterogeneous data is
still lacking. To resolve the theoretical dilemma of SFL, we establish sharp
convergence guarantees for SFL on heterogeneous data with both upper and lower
bounds. Specifically, we derive the upper bounds for strongly convex, general
convex and non-convex objective functions, and construct the matching lower
bounds for the strongly convex and general convex objective functions. Then, we
compare the upper bounds of SFL with those of PFL, showing that SFL outperforms
PFL (at least, when the level of heterogeneity is relatively high).
Experimental results on quadratic functions and real data sets validate the
counterintuitive comparison result.
