  Kernel ridge regression (KRR) is a fundamental computational tool, appearing
in problems that range from computational chemistry to health analytics, with a
particular interest due to its starring role in Gaussian process regression.
However, it is challenging to scale KRR solvers to large datasets: with $n$
training points, a direct solver (i.e., Cholesky decomposition) uses $O(n^2)$
storage and $O(n^3)$ flops. Iterative methods for KRR, such as preconditioned
conjugate gradient (PCG), avoid the cubic scaling of direct solvers and often
use low-rank preconditioners; a rank $r$ preconditioner uses $O(rn)$ storage
and each iteration requires $O(n^2)$ flops. To reduce the storage and iteration
complexity of iterative solvers for KRR, we propose ASkotch
($\textbf{A}$ccelerated $\textbf{s}$calable $\textbf{k}$ernel
$\textbf{o}$p$\textbf{t}$imization using block $\textbf{c}$oordinate descent
with $\textbf{H}$essian preconditioning). For a given block size $|b| << n$,
each iteration of ASkotch uses $O(r|b| + n)$ storage and $O(n|b|)$ flops, so
ASkotch scales better than Cholesky decomposition and PCG. We prove that
ASkotch obtains linear convergence to the optimum, with the convergence rate
depending on the square roots of the $\textit{preconditioned}$ block condition
numbers. Furthermore, we solve KRR problems that were considered to be
impossibly large while using limited computational resources: we show that
ASkotch outperforms PCG methods with respect to generalization error on
large-scale KRR (up to $n = 10^8$) and KRR classification tasks (up to $n =
10^7$) while running each of our experiments on $\textit{a single 12 GB Titan V
GPU}$. Our work opens up the possibility of as-yet-unimagined applications of
KRR across a wide range of disciplines.
