  In this paper we study the differentially private Empirical Risk Minimization
(ERM) problem in different settings. For smooth (strongly) convex loss function
with or without (non)-smooth regularization, we give algorithms that achieve
either optimal or near optimal utility bounds with less gradient complexity
compared with previous work. For ERM with smooth convex loss function in
high-dimensional ($p\gg n$) setting, we give an algorithm which achieves the
upper bound with less gradient complexity than previous ones. At last, we
generalize the expected excess empirical risk from convex loss functions to
non-convex ones satisfying the Polyak-Lojasiewicz condition and give a tighter
upper bound on the utility than the one in \cite{ijcai2017-548}.
