  Serving ML prediction pipelines spanning multiple models and hardware
accelerators is a key challenge in production machine learning. Optimally
configuring these pipelines to meet tight end-to-end latency goals is
complicated by the interaction between model batch size, the choice of hardware
accelerator, and variation in the query arrival process.
  In this paper we introduce InferLine, a system which provisions and manages
the individual stages of prediction pipelines to meet end-to-end tail latency
constraints while minimizing cost. InferLine consists of a low-frequency
combinatorial planner and a high-frequency auto-scaling tuner. The
low-frequency planner leverages stage-wise profiling, discrete event
simulation, and constrained combinatorial search to automatically select
hardware type, replication, and batching parameters for each stage in the
pipeline. The high-frequency tuner uses network calculus to auto-scale each
stage to meet tail latency goals in response to changes in the query arrival
process. We demonstrate that InferLine outperforms existing approaches by up to
7.6x in cost while achieving up to 34.5x lower latency SLO miss rate on
realistic workloads and generalizes across state-of-the-art model serving
frameworks.
