  The privacy concerns associated with the use of Large Language Models (LLMs)
have grown recently with the development of LLMs such as ChatGPT. Differential
Privacy (DP) techniques are explored in existing work to mitigate their privacy
risks at the cost of generalization degradation. Our paper reveals that the
flatness of DP-trained models' loss landscape plays an essential role in the
trade-off between their privacy and generalization. We further propose a
holistic framework to enforce appropriate weight flatness, which substantially
improves model generalization with competitive privacy preservation. It
innovates from three coarse-to-grained levels, including perturbation-aware
min-max optimization on model weights within a layer, flatness-guided sparse
prefix-tuning on weights across layers, and weight knowledge distillation
between DP \& non-DP weights copies. Comprehensive experiments of both
black-box and white-box scenarios are conducted to demonstrate the
effectiveness of our proposal in enhancing generalization and maintaining DP
characteristics. For instance, on text classification dataset QNLI, DP-Flat
achieves similar performance with non-private full fine-tuning but with DP
guarantee under privacy budget $\epsilon=3$, and even better performance given
higher privacy budgets. Codes are provided in the supplement.
