  Since random initial models in Federated Learning (FL) can easily result in
unregulated Stochastic Gradient Descent (SGD) processes, existing FL methods
greatly suffer from both slow convergence and poor accuracy, especially for
non-IID scenarios. To address this problem, we propose a novel FL method named
CyclicFL, which can quickly derive effective initial models to guide the SGD
processes, thus improving the overall FL training performance. Based on the
concept of Continual Learning (CL), we prove that CyclicFL approximates
existing centralized pre-training methods in terms of classification and
prediction performance. Meanwhile, we formally analyze the significance of data
consistency between the pre-training and training stages of CyclicFL, showing
the limited Lipschitzness of loss for the pre-trained models by CyclicFL.
Unlike traditional centralized pre-training methods that require public proxy
data, CyclicFL pre-trains initial models on selected clients cyclically without
exposing their local data. Therefore, they can be easily integrated into any
security-critical FL methods. Comprehensive experimental results show that
CyclicFL can not only improve the classification accuracy by up to 16.21%, but
also significantly accelerate the overall FL training processes.
