  Normalization is an important but understudied challenge in privacy-related
application domains such as federated learning (FL), differential privacy (DP),
and differentially private federated learning (DP-FL). While the unsuitability
of batch normalization for these domains has already been shown, the impact of
other normalization methods on the performance of federated or differentially
private models is not well-known. To address this, we draw a performance
comparison among layer normalization (LayerNorm), group normalization
(GroupNorm), and the recently proposed kernel normalization (KernelNorm) in FL,
DP, and DP-FL settings. Our results indicate LayerNorm and GroupNorm provide no
performance gain compared to the baseline (i.e. no normalization) for shallow
models in FL and DP. They, on the other hand, considerably enhance the
performance of shallow models in DP-FL and deeper models in FL and DP.
KernelNorm, moreover, significantly outperforms its competitors in terms of
accuracy and convergence rate (or communication efficiency) for both shallow
and deeper models in all considered learning environments. Given these key
observations, we propose a kernel normalized ResNet architecture called
KNResNet-13 for differentially private learning. Using the proposed
architecture, we provide new state-of-the-art accuracy values on the CIFAR-10
and Imagenette datasets, when trained from scratch.
