  We present ALTO, a network orchestrator for efficiently serving compound AI
systems such as pipelines of language models. ALTO achieves high throughput and
low latency by taking advantage of an optimization opportunity specific to
generative language models: streaming intermediate outputs. As language models
produce outputs token by token, ALTO exposes opportunities to stream
intermediate outputs between stages when possible. We highlight two new
challenges of correctness and load balancing which emerge when streaming
intermediate data across distributed pipeline stage instances. We also motivate
the need for an aggregation-aware routing interface and distributed
prompt-aware scheduling to address these challenges. We demonstrate the impact
of ALTO's partial output streaming on a complex chatbot verification pipeline,
increasing throughput by up to 3x for a fixed latency target of 4 seconds /
request while also reducing tail latency by 1.8x compared to a baseline serving
approach.
