  We revisit the problem of linear regression under a differential privacy
constraint. By consolidating existing pieces in the literature, we clarify the
correct dependence of the feature, label and coefficient domains in the
optimization error and estimation error, hence revealing the delicate price of
differential privacy in statistical estimation and statistical learning.
Moreover, we propose simple modifications of two existing DP algorithms: (a)
posterior sampling, (b) sufficient statistics perturbation, and show that they
can be upgraded into **adaptive** algorithms that are able to exploit
data-dependent quantities and behave nearly optimally **for every instance**.
Extensive experiments are conducted on both simulated data and real data, which
conclude that both AdaOPS and AdaSSP outperform the existing techniques on
nearly all 36 data sets that we test on.
