  The development of real-world Large Language Models (LLMs) necessitates
checkpointing of training states in persistent storage to mitigate potential
software and hardware failures, as well as to facilitate checkpoint
transferring within the training pipeline and across various tasks. Due to the
immense size of LLMs, saving and loading checkpoints often incur intolerable
minute-level stalls, significantly diminishing training efficiency. Besides,
when transferring checkpoints across tasks, checkpoint resharding, defined as
loading checkpoints into parallel configurations differing from those used for
saving, is often required according to the characteristics and resource quota
of specific tasks. Previous checkpointing systems [16,3,33,6] assume consistent
parallel configurations, failing to address the complexities of checkpoint
transformation during resharding. Furthermore, in the industry platform,
developers create checkpoints from different training frameworks[23,36,21,11],
each with its own unique storage and I/O logic. This diversity complicates the
implementation of unified checkpoint management and optimization. To address
these challenges, we introduce ByteCheckpoint, a PyTorch-native multi-framework
LLM checkpointing system that supports automatic online checkpoint resharding.
ByteCheckpoint employs a data/metadata disaggregated storage architecture,
decoupling checkpoint storage from the adopted parallelism strategies and
training frameworks. We design an efficient asynchronous tensor merging
technique to settle the irregular tensor sharding problem and propose several
I/O performance optimizations to significantly enhance the efficiency of
checkpoint saving and loading. Experimental results demonstrate
ByteCheckpoint's substantial advantages in reducing checkpoint saving (by up to
529.22X) and loading (by up to 3.51X) costs, compared to baseline methods.
