  Differential privacy (DP) provides a formal privacy guarantee that prevents
adversaries with access to machine learning models from extracting information
about individual training points. Differentially private stochastic gradient
descent (DPSGD) is the most popular training method with differential privacy
in image recognition. However, existing DPSGD schemes lead to significant
performance degradation, which prevents the application of differential
privacy. In this paper, we propose a simulated annealing-based differentially
private stochastic gradient descent scheme (SA-DPSGD) which accepts a candidate
update with a probability that depends both on the update quality and on the
number of iterations. Through this random update screening, we make the
differentially private gradient descent proceed in the right direction in each
iteration, and result in a more accurate model finally. In our experiments,
under the same hyperparameters, our scheme achieves test accuracies 98.35%,
87.41% and 60.92% on datasets MNIST, FashionMNIST and CIFAR10, respectively,
compared to the state-of-the-art result of 98.12%, 86.33% and 59.34%. Under the
freely adjusted hyperparameters, our scheme achieves even higher accuracies,
98.89%, 88.50% and 64.17%. We believe that our method has a great contribution
for closing the accuracy gap between private and non-private image
classification.
