  As the size of deep learning models gets larger and larger, training takes
longer time and more resources, making fault tolerance more and more critical.
Existing state-of-the-art methods like CheckFreq and Elastic Horovod need to
back up a copy of the model state (i.e., parameters and optimizer states) in
memory, which is costly for large models and leads to non-trivial overhead.
This paper presents SWIFT, a novel recovery design for distributed deep neural
network training that significantly reduces the failure recovery overhead
without affecting training throughput and model accuracy. Instead of making an
additional copy of the model state, SWIFT resolves the inconsistencies of the
model state caused by the failure and exploits the replicas of the model state
in data parallelism for failure recovery. We propose a logging-based approach
when replicas are unavailable, which records intermediate data and replays the
computation to recover the lost state upon a failure. The re-computation is
distributed across multiple machines to accelerate failure recovery further. We
also log intermediate data selectively, exploring the trade-off between
recovery time and intermediate data storage overhead. Evaluations show that
SWIFT significantly reduces the failure recovery time and achieves similar or
better training throughput during failure-free execution compared to
state-of-the-art methods without degrading final model accuracy. SWIFT can also
achieve up to 1.16x speedup in total training time compared to state-of-the-art
methods.
