  Random matrix theory has become a widely useful tool in high-dimensional
statistics and theoretical machine learning. However, random matrix theory is
largely focused on the proportional asymptotics in which the number of columns
grows proportionally to the number of rows of the data matrix. This is not
always the most natural setting in statistics where columns correspond to
covariates and rows to samples. With the objective to move beyond the
proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least
squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector
and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow
the feature vector to be high-dimensional, or even infinite-dimensional, in
which case it belongs to a separable Hilbert space, and assume either $z_i :=
\Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex
concentration property. Within this setting, we establish non-asymptotic bounds
that approximate the bias and variance of ridge regression in terms of the bias
and variance of an `equivalent' sequence model (a regression model with
diagonal design matrix). The approximation is up to multiplicative factors
bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such
an approximation result was known only in the proportional regime and only up
to additive errors: in particular, it did not allow to characterize the
behavior of the excess risk when this converges to $0$. Our general theory
recovers earlier results in the proportional regime (with better error rates).
As a new application, we obtain a completely explicit and sharp
characterization of ridge regression for Hilbert covariates with regularly
varying spectrum. Finally, we analyze the overparametrized near-interpolation
setting and obtain sharp `benign overfitting' guarantees.
