  Given $n$ vectors $\mathbf{x}_i\in \mathbb{R}^d$, we want to fit a linear
regression model for noisy labels $y_i\in\mathbb{R}$. The ridge estimator is a
classical solution to this problem. However, when labels are expensive, we are
forced to select only a small subset of vectors $\mathbf{x}_i$ for which we
obtain the labels $y_i$. We propose a new procedure for selecting the subset of
vectors, such that the ridge estimator obtained from that subset offers strong
statistical guarantees in terms of the mean squared prediction error over the
entire dataset of $n$ labeled vectors. The number of labels needed is
proportional to the statistical dimension of the problem which is often much
smaller than $d$. Our method is an extension of a joint subsampling procedure
called volume sampling. A second major contribution is that we speed up volume
sampling so that it is essentially as efficient as leverage scores, which is
the main i.i.d. subsampling procedure for this task. Finally, we show
theoretically and experimentally that volume sampling has a clear advantage
over any i.i.d. sampling when labels are expensive.
