  We provide computationally efficient, differentially private algorithms for
the classical regression settings of Least Squares Fitting, Binary Regression
and Linear Regression with unbounded covariates. Prior to our work, privacy
constraints in such regression settings were studied under strong a priori
bounds on covariates. We consider the case of Gaussian marginals and extend
recent differentially private techniques on mean and covariance estimation
(Kamath et al., 2019; Karwa and Vadhan, 2018) to the sub-gaussian regime. We
provide a novel technical analysis yielding differentially private algorithms
for the above classical regression settings. Through the case of Binary
Regression, we capture the fundamental and widely-studied models of logistic
regression and linearly-separable SVMs, learning an unbiased estimate of the
true regression vector, up to a scaling factor.
