  Differentially private federated learning (DP-FL) is a promising technique
for collaborative model training while ensuring provable privacy for clients.
However, optimizing the tradeoff between privacy and accuracy remains a
critical challenge. To our best knowledge, we propose the first DP-FL framework
(namely UDP-FL), which universally harmonizes any randomization mechanism
(e.g., an optimal one) with the Gaussian Moments Accountant (viz. DP-SGD) to
significantly boost accuracy and convergence. Specifically, UDP-FL demonstrates
enhanced model performance by mitigating the reliance on Gaussian noise. The
key mediator variable in this transformation is the R\'enyi Differential
Privacy notion, which is carefully used to harmonize privacy budgets. We also
propose an innovative method to theoretically analyze the convergence for DP-FL
(including our UDP-FL ) based on mode connectivity analysis. Moreover, we
evaluate our UDP-FL through extensive experiments benchmarked against
state-of-the-art (SOTA) methods, demonstrating superior performance on both
privacy guarantees and model performance. Notably, UDP-FL exhibits substantial
resilience against different inference attacks, indicating a significant
advance in safeguarding sensitive data in federated learning environments.
