  Nowadays, deep learning methods with large-scale datasets can produce
clinically useful models for computer-aided diagnosis. However, the privacy and
ethical concerns are increasingly critical, which make it difficult to collect
large quantities of data from multiple institutions. Federated Learning (FL)
provides a promising decentralized solution to train model collaboratively by
exchanging client models instead of private data. However, the server
aggregation of existing FL methods is observed to degrade the model performance
in real-world medical FL setting, which is termed as retrogress. To address
this problem, we propose a personalized retrogress-resilient framework to
produce a superior personalized model for each client. Specifically, we devise
a Progressive Fourier Aggregation (PFA) at the server to achieve more stable
and effective global knowledge gathering by integrating client models from
low-frequency to high-frequency gradually. Moreover, with an introduced deputy
model to receive the aggregated server model, we design a Deputy-Enhanced
Transfer (DET) strategy at the client and conduct three steps of
Recover-Exchange-Sublimate to ameliorate the personalized local model by
transferring the global knowledge smoothly. Extensive experiments on real-world
dermoscopic FL dataset prove that our personalized retrogress-resilient
framework outperforms state-of-the-art FL methods, as well as the
generalization on an out-of-distribution cohort. The code and dataset are
available at https://github.com/CityU-AIM-Group/PRR-FL.
