  In the era of deep learning (DL), convolutional neural networks (CNNs), and
large language models (LLMs), machine learning (ML) models are becoming
increasingly complex, demanding significant computational resources for both
inference and training stages. To address this challenge, distributed learning
has emerged as a crucial approach, employing parallelization across various
devices and environments. This survey explores the landscape of distributed
learning, encompassing cloud and edge settings. We delve into the core concepts
of data and model parallelism, examining how models are partitioned across
different dimensions and layers to optimize resource utilization and
performance. We analyze various partitioning schemes for different layer types,
including fully connected, convolutional, and recurrent layers, highlighting
the trade-offs between computational efficiency, communication overhead, and
memory constraints. This survey provides valuable insights for future research
and development in this rapidly evolving field by comparing and contrasting
distributed learning approaches across diverse contexts.
