  Modern machine learning algorithms aim to extract fine-grained information
from data to provide accurate predictions, which often conflicts with the goal
of privacy protection. This paper addresses the practical and theoretical
importance of developing privacy-preserving machine learning algorithms that
ensure good performance while preserving privacy. In this paper, we focus on
the privacy and utility (measured by excess risk bounds) performances of
differentially private stochastic gradient descent (SGD) algorithms in the
setting of stochastic convex optimization. Specifically, we examine the
pointwise problem in the low-noise setting for which we derive sharper excess
risk bounds for the differentially private SGD algorithm. In the pairwise
learning setting, we propose a simple differentially private SGD algorithm
based on gradient perturbation. Furthermore, we develop novel utility bounds
for the proposed algorithm, proving that it achieves optimal excess risk rates
even for non-smooth losses. Notably, we establish fast learning rates for
privacy-preserving pairwise learning under the low-noise condition, which is
the first of its kind.
