  Datacenter-scale AI training clusters consisting of thousands of
domain-specific accelerators (DSA) are used to train increasingly-complex deep
learning models. These clusters rely on a data storage and ingestion (DSI)
pipeline, responsible for storing exabytes of training data and serving it at
tens of terabytes per second. As DSAs continue to push training efficiency and
throughput, the DSI pipeline is becoming the dominating factor that constrains
the overall training performance and capacity. Innovations that improve the
efficiency and performance of DSI systems and hardware are urgent, demanding a
deep understanding of DSI characteristics and infrastructure at scale.
  This paper presents Meta's end-to-end DSI pipeline, composed of a central
data warehouse built on distributed storage and a Data PreProcessing Service
that scales to eliminate data stalls. We characterize how hundreds of models
are collaboratively trained across geo-distributed datacenters via diverse and
continuous training jobs. These training jobs read and heavily filter massive
and evolving datasets, resulting in popular features and samples used across
training jobs. We measure the intense network, memory, and compute resources
required by each training job to preprocess samples during training. Finally,
we synthesize key takeaways based on our production infrastructure
characterization. These include identifying hardware bottlenecks, discussing
opportunities for heterogeneous DSI hardware, motivating research in datacenter
scheduling and benchmark datasets, and assimilating lessons learned in
optimizing DSI infrastructure.
