  A recent study by De et al. (2022) has reported that large-scale
representation learning through pre-training on a public dataset significantly
enhances differentially private (DP) learning in downstream tasks, despite the
high dimensionality of the feature space. To theoretically explain this
phenomenon, we consider the setting of a layer-peeled model in representation
learning, which results in interesting phenomena related to learned features in
deep learning and transfer learning, known as Neural Collapse (NC).
  Within the framework of NC, we establish an error bound indicating that the
misclassification error is independent of dimension when the distance between
actual features and the ideal ones is smaller than a threshold. Additionally,
the quality of the features in the last layer is empirically evaluated under
different pre-trained models within the framework of NC, showing that a more
powerful transformer leads to a better feature representation. Furthermore, we
reveal that DP fine-tuning is less robust compared to fine-tuning without DP,
particularly in the presence of perturbations. These observations are supported
by both theoretical analyses and experimental evaluation. Moreover, to enhance
the robustness of DP fine-tuning, we suggest several strategies, such as
feature normalization or employing dimension reduction methods like Principal
Component Analysis (PCA). Empirically, we demonstrate a significant improvement
in testing accuracy by conducting PCA on the last-layer features.
