  Classical regression has a simple geometric description in terms of a
projection of the training labels onto the column space of the design matrix.
However, for over-parameterized models -- where the number of fit parameters is
large enough to perfectly fit the training data -- this picture becomes
uninformative. Here, we present an alternative geometric interpretation of
regression that applies to both under- and over-parameterized models. Unlike
the classical picture which takes place in the space of training labels, our
new picture resides in the space of input features. This new feature-based
perspective provides a natural geometric interpretation of the double-descent
phenomenon in the context of bias and variance, explaining why it can occur
even in the absence of label noise. Furthermore, we show that adversarial
perturbations -- small perturbations to the input features that result in large
changes in label values -- are a generic feature of biased models, arising from
the underlying geometry. We demonstrate these ideas by analyzing three minimal
models for over-parameterized linear least squares regression: without basis
functions (input features equal model features) and with linear or nonlinear
basis functions (two-layer neural networks with linear or nonlinear activation
functions, respectively).
