  Training large Deep Neural Network (DNN) models requires thousands of GPUs
for days or weeks at a time. At these scales, failures are frequent and can
have a big impact on training throughput. Restoring performance using spare GPU
servers becomes increasingly expensive as models grow. SlipStream is a system
for efficient DNN training in the presence of failures, without using spare
servers. It exploits the functional redundancy inherent in distributed training
systems -- servers hold the same model parameters across data-parallel groups
-- as well as the bubbles in the pipeline schedule within each data-parallel
group. SlipStream dynamically re-routes the work of a failed server to its
data-parallel peers, ensuring continuous training despite multiple failures.
However, re-routing work leads to imbalances across pipeline stages that
degrades training throughput. SlipStream introduces two optimizations that
allow re-routed work to execute within bubbles of the original pipeline
schedule. First, it decouples the backward pass computation into two phases.
Second, it staggers the execution of the optimizer step across pipeline stages.
Combined, these optimizations enable schedules that minimize or even eliminate
training throughput degradation during failures. We describe a prototype for
SlipStream and show that it achieves high training throughput under multiple
failures, outperforming recent proposals for fault-tolerant training such as
Oobleck and Bamboo by up to 1.46x and 1.64x, respectively.
