  We study the residual bootstrap (RB) method in the context of
high-dimensional linear regression. Specifically, we analyze the distributional
approximation of linear contrasts $c^{\top} (\hat{\beta}_{\rho}-\beta)$, where
$\hat{\beta}_{\rho}$ is a ridge-regression estimator. When regression
coefficients are estimated via least squares, classical results show that RB
consistently approximates the laws of contrasts, provided that $p\ll n$, where
the design matrix is of size $n\times p$. Up to now, relatively little work has
considered how additional structure in the linear model may extend the validity
of RB to the setting where $p/n\asymp 1$. In this setting, we propose a version
of RB that resamples residuals obtained from ridge regression. Our main
structural assumption on the design matrix is that it is nearly low rank --- in
the sense that its singular values decay according to a power-law profile.
Under a few extra technical assumptions, we derive a simple criterion for
ensuring that RB consistently approximates the law of a given contrast. We then
specialize this result to study confidence intervals for mean response values
$X_i^{\top} \beta$, where $X_i^{\top}$ is the $i$th row of the design. More
precisely, we show that conditionally on a Gaussian design with near low-rank
structure, RB simultaneously approximates all of the laws
$X_i^{\top}(\hat{\beta}_{\rho}-\beta)$, $i=1,\dots,n$. This result is also
notable as it imposes no sparsity assumptions on $\beta$. Furthermore, since
our consistency results are formulated in terms of the Mallows (Kantorovich)
metric, the existence of a limiting distribution is not required.
