  The Ridgeless minimum $\ell_2$-norm interpolator in overparametrized linear
regression has attracted considerable attention in recent years. While it seems
to defy the conventional wisdom that overfitting leads to poor prediction,
recent research reveals that its norm minimizing property induces an `implicit
regularization' that helps prediction in spite of interpolation. This renders
the Ridgeless interpolator a theoretically tractable proxy that offers useful
insights into the mechanisms of modern machine learning methods.
  This paper takes a different perspective that aims at understanding the
precise stochastic behavior of the Ridgeless interpolator as a statistical
estimator. Specifically, we characterize the distribution of the Ridgeless
interpolator in high dimensions, in terms of a Ridge estimator in an associated
Gaussian sequence model with positive regularization, which plays the role of
the prescribed implicit regularization in the context of prediction risk. Our
distributional characterizations hold for general random designs and extend
uniformly to positively regularized Ridge estimators. As a demonstration of the
analytic power of these characterizations, we derive approximate formulae for a
general class of weighted $\ell_q$ risks for Ridge(less) estimators that were
previously available only for $\ell_2$. Our theory also provides certain
further conceptual reconciliation with the conventional wisdom: given any data
covariance, a certain amount of regularization in Ridge regression remains
beneficial for `most' signals across various statistical tasks including
prediction, estimation and inference, as long as the noise level is
non-trivial. Surprisingly, optimal tuning can be achieved simultaneously for
all the designated statistical tasks by a single generalized or $k$-fold
cross-validation scheme, despite being designed specifically for tuning
prediction risk.
