  Efficient construction of checkpoints/snapshots is a critical tool for
training and diagnosing deep learning models. In this paper, we propose a lossy
compression scheme for checkpoint constructions (called LC-Checkpoint).
LC-Checkpoint simultaneously maximizes the compression rate and optimizes the
recovery speed, under the assumption that SGD is used to train the model.
LC-Checkpointuses quantization and priority promotion to store the most crucial
information for SGD to recover, and then uses a Huffman coding to leverage the
non-uniform distribution of the gradient scales. Our extensive experiments show
that LC-Checkpoint achieves a compression rate up to $28\times$ and recovery
speedup up to $5.77\times$ over a state-of-the-art algorithm (SCAR).
