  The expansion of IoT devices and the demands of Deep Learning have
highlighted significant challenges in Distributed Deep Learning (DDL) systems.
Parallel Split Learning (PSL) has emerged as a promising derivative of Split
Learning that is well suited for distributed learning on resource-constrained
devices. However, PSL faces several obstacles, such as large effective batch
sizes, non-IID data distributions, and the straggler effect. We view these
issues as a sampling dilemma and propose to address them by orchestrating the
mini-batch sampling process on the server side. We introduce the Uniform Global
Sampling (UGS) method to decouple the effective batch size from the number of
clients and reduce mini-batch deviation in non-IID settings. To address the
straggler effect, we introduce the Latent Dirichlet Sampling (LDS) method,
which generalizes UGS to balance the trade-off between batch deviation and
training time. Our simulations reveal that our proposed methods enhance model
accuracy by up to 34.1% in non-IID settings and reduce the training time in the
presence of stragglers by up to 62%. In particular, LDS effectively mitigates
the straggler effect without compromising model accuracy or adding significant
computational overhead compared to UGS. Our results demonstrate the potential
of our methods as a promising solution for DDL in real applications.
