  With rapidly increasing distributed deep learning workloads in large-scale
data centers, efficient distributed deep learning framework strategies for
resource allocation and workload scheduling have become the key to
high-performance deep learning. The large-scale environment with large volumes
of datasets, models, and computational and communication resources raises
various unique challenges for resource allocation and workload scheduling in
distributed deep learning, such as scheduling complexity, resource and workload
heterogeneity, and fault tolerance. To uncover these challenges and
corresponding solutions, this survey reviews the literature, mainly from 2019
to 2024, on efficient resource allocation and workload scheduling strategies
for large-scale distributed DL. We explore these strategies by focusing on
various resource types, scheduling granularity levels, and performance goals
during distributed training and inference processes. We highlight critical
challenges for each topic and discuss key insights of existing technologies. To
illustrate practical large-scale resource allocation and workload scheduling in
real distributed deep learning scenarios, we use a case study of training large
language models. This survey aims to encourage computer science, artificial
intelligence, and communications researchers to understand recent advances and
explore future research directions for efficient framework strategies for
large-scale distributed deep learning.
