  Effective training of today's large language models (LLMs) depends on large
batches and long sequences for throughput and accuracy. To handle
variable-length sequences on hardware accelerators, it is common practice to
introduce padding tokens, so that all sequences in a batch have the same
length. We show in this paper that the variation in sequence lengths in common
NLP datasets is such that up to 50% of all tokens can be padding. In less
common, but not extreme, cases (e.g. GLUE-cola with sequence length 128), the
ratio is up to 89%. Existing methods to address the resulting inefficiency are
complicated by the need to avoid cross-contamination in self-attention, by a
reduction in accuracy when sequence ordering information is lost, or by
customized kernel implementations only valid for specific accelerators. This
paper introduces a new formalization of sequence packing in the context of the
well-studied bin packing problem, and presents new algorithms based on this
formulation which, for example, confer a 2x speedup for phase 2 pre-training in
BERT. We show how existing models can be adapted to ensure mathematical
equivalence between the original and packed models, meaning that packed models
can be trained with existing pre-training and fine-tuning practices.
