  Deep learning research has uncovered the phenomenon of benign overfitting for
overparameterized statistical models, which has drawn significant theoretical
interest in recent years. Given its simplicity and practicality, the ordinary
least squares (OLS) interpolator has become essential to gain foundational
insights into this phenomenon. While properties of OLS are well established in
classical, underparameterized settings, its behavior in high-dimensional,
overparameterized regimes is less explored (unlike for ridge or lasso
regression) though significant progress has been made of late. We contribute to
this growing literature by providing fundamental algebraic and statistical
results for the minimum $\ell_2$-norm OLS interpolator. In particular, we
provide algebraic equivalents of (i) the leave-$k$-out residual formula, (ii)
Cochran's formula, and (iii) the Frisch-Waugh-Lovell theorem in the
overparameterized regime. These results aid in understanding the OLS
interpolator's ability to generalize and have substantive implications for
causal inference. Under the Gauss-Markov model, we present statistical results
such as an extension of the Gauss-Markov theorem and an analysis of variance
estimation under homoskedastic errors for the overparameterized regime. To
substantiate our theoretical contributions, we conduct simulations that further
explore the stochastic properties of the OLS interpolator.
