  Traditionally, the random noise is equally injected when training with
different data instances in the field of differential privacy (DP). In this
paper, we first give sharper excess risk bounds of DP stochastic gradient
descent (SGD) method. Considering most of the previous methods are under convex
conditions, we use Polyak-{\L}ojasiewicz condition to relax it in this paper.
Then, after observing that different training data instances affect the machine
learning model to different extent, we consider the heterogeneity of training
data and attempt to improve the performance of DP-SGD from a new perspective.
Specifically, by introducing the influence function (IF), we quantitatively
measure the contributions of various training data on the final machine
learning model. If the contribution made by a single data instance is so little
that attackers cannot infer anything from the model, we do not add noise when
training with it. Based on this observation, we design a `Performance
Improving' DP-SGD algorithm: PIDP-SGD. Theoretical and experimental results
show that our proposed PIDP-SGD improves the performance significantly.
