  Checkpoints play an important role in training long running machine learning
(ML) models. Checkpoints take a snapshot of an ML model and store it in a
non-volatile memory so that they can be used to recover from failures to ensure
rapid training progress. In addition, they are used for online training to
improve inference prediction accuracy with continuous learning. Given the large
and ever increasing model sizes, checkpoint frequency is often bottlenecked by
the storage write bandwidth and capacity. When checkpoints are maintained on
remote storage, as is the case with many industrial settings, they are also
bottlenecked by network bandwidth. We present Check-N-Run, a scalable
checkpointing system for training large ML models at Facebook. While
Check-N-Run is applicable to long running ML jobs, we focus on checkpointing
recommendation models which are currently the largest ML models with Terabytes
of model size. Check-N-Run uses two primary techniques to address the size and
bandwidth challenges. First, it applies incremental checkpointing, which tracks
and checkpoints the modified part of the model. Incremental checkpointing is
particularly valuable in the context of recommendation models where only a
fraction of the model (stored as embedding tables) is updated on each
iteration. Second, Check-N-Run leverages quantization techniques to
significantly reduce the checkpoint size, without degrading training accuracy.
These techniques allow Check-N-Run to reduce the required write bandwidth by
6-17x and the required capacity by 2.5-8x on real-world models at Facebook, and
thereby significantly improve checkpoint capabilities while reducing the total
cost of ownership.
