  We study the problem of Differentially Private Stochastic Convex Optimization
(DP-SCO) with heavy-tailed data. Specifically, we focus on the $\ell_1$-norm
linear regression in the $\epsilon$-DP model. While most of the previous work
focuses on the case where the loss function is Lipschitz, here we only need to
assume the variates has bounded moments. Firstly, we study the case where the
$\ell_2$ norm of data has bounded second order moment. We propose an algorithm
which is based on the exponential mechanism and show that it is possible to
achieve an upper bound of $\tilde{O}(\sqrt{\frac{d}{n\epsilon}})$ (with high
probability). Next, we relax the assumption to bounded $\theta$-th order moment
with some $\theta\in (1, 2)$ and show that it is possible to achieve an upper
bound of $\tilde{O}(({\frac{d}{n\epsilon}})^\frac{\theta-1}{\theta})$. Our
algorithms can also be extended to more relaxed cases where only each
coordinate of the data has bounded moments, and we can get an upper bound of
$\tilde{O}({\frac{d}{\sqrt{n\epsilon}}})$ and
$\tilde{O}({\frac{d}{({n\epsilon})^\frac{\theta-1}{\theta}}})$ in the second
and $\theta$-th moment case respectively.
