  Modern applications require methods that are computationally feasible on
large datasets but also preserve statistical efficiency. Frequently, these two
concerns are seen as contradictory: approximation methods that enable
computation are assumed to degrade statistical performance relative to exact
methods. In applied mathematics, where much of the current theoretical work on
approximation resides, the inputs are considered to be observed exactly. The
prevailing philosophy is that while the exact problem is, regrettably,
unsolvable, any approximation should be as small as possible. However, from a
statistical perspective, an approximate or regularized solution may be
preferable to the exact one. Regularization formalizes a trade-off between
fidelity to the data and adherence to prior knowledge about the data-generating
process such as smoothness or sparsity. The resulting estimator tends to be
more useful, interpretable, and suitable as an input to other methods.
  In this paper, we propose new methodology for estimation and prediction under
a linear model borrowing insights from the approximation literature. We explore
these procedures from a statistical perspective and find that in many cases
they improve both computational and statistical performance.
