  Kernel matrix-vector multiplication (KMVM) is a foundational operation in
machine learning and scientific computing. However, as KMVM tends to scale
quadratically in both memory and time, applications are often limited by these
computational constraints. In this paper, we propose a novel approximation
procedure coined \textit{Faster-Fast and Free Memory Method} ($\fthreem$) to
address these scaling issues of KMVM for tall~($10^8\sim 10^9$) and
skinny~($D\leq7$) data. Extensive experiments demonstrate that $\fthreem$ has
empirical \emph{linear time and memory} complexity with a relative error of
order $10^{-3}$ and can compute a full KMVM for a billion points \emph{in under
a minute} on a high-end GPU, leading to a significant speed-up in comparison to
existing CPU methods. We demonstrate the utility of our procedure by applying
it as a drop-in for the state-of-the-art GPU-based linear solver FALKON,
\emph{improving speed 1.5-5.5 times} at the cost of $<1\%$ drop in accuracy. We
further demonstrate competitive results on \emph{Gaussian Process regression}
coupled with significant speedups on a variety of real-world datasets.
