  Systems for ML inference are widely deployed today, but they typically
optimize ML inference workloads using techniques designed for conventional data
serving workloads and miss critical opportunities to leverage the statistical
nature of ML. In this paper, we present Willump, an optimizer for ML inference
that introduces two statistically-motivated optimizations targeting ML
applications whose performance bottleneck is feature computation. First,
Willump automatically cascades feature computation for classification queries:
Willump classifies most data inputs using only high-value, low-cost features
selected through empirical observations of ML model performance, improving
query performance by up to 5x without statistically significant accuracy loss.
Second, Willump accurately approximates ML top-K queries, discarding
low-scoring inputs with an automatically constructed approximate model and then
ranking the remainder with a more powerful model, improving query performance
by up to 10x with minimal accuracy loss. Willump automatically tunes these
optimizations' parameters to maximize query performance while meeting an
accuracy target. Moreover, Willump complements these statistical optimizations
with compiler optimizations to automatically generate fast inference code for
ML applications. We show that Willump improves the end-to-end performance of
real-world ML inference pipelines curated from major data science competitions
by up to 16x without statistically significant loss of accuracy.
