  Deep learning (DL) has transformed applications in a variety of domains,
including computer vision, natural language processing, and tabular data
analysis. The search for improved DL model accuracy has led practitioners to
explore increasingly large neural architectures, with some recent Transformer
models spanning hundreds of billions of learnable parameters. These designs
have introduced new scale-driven systems challenges for the DL space, such as
memory bottlenecks, poor runtime efficiency, and high costs of model
development. Efforts to address these issues have explored techniques such as
parallelization of neural architectures, spilling data across the memory
hierarchy, and memory-efficient data representations. This survey will explore
the large-model training systems landscape, highlighting key challenges and the
various techniques that have been used to address them.
