  A major challenge in applying differential privacy to training deep neural
network models is scalability.The widely-used training algorithm,
differentially private stochastic gradient descent (DP-SGD), struggles with
training moderately-sized neural network models for a value of epsilon
corresponding to a high level of privacy protection. In this paper, we explore
the idea of dimensionality reduction inspired by neural network pruning to
improve the scalability of DP-SGD. We study the interplay between neural
network pruning and differential privacy, through the two modes of parameter
updates. We call the first mode, parameter freezing, where we pre-prune the
network and only update the remaining parameters using DP-SGD. We call the
second mode, parameter selection, where we select which parameters to update at
each step of training and update only those selected using DP-SGD. In these
modes, we use public data for freezing or selecting parameters to avoid privacy
loss incurring in these steps. Naturally, the closeness between the private and
public data plays an important role in the success of this paradigm. Our
experimental results demonstrate how decreasing the parameter space improves
differentially private training. Moreover, by studying two popular forms of
pruning which do not rely on gradients and do not incur an additional privacy
loss, we show that random selection performs on par with magnitude-based
selection when it comes to DP-SGD training.
