  Training large models requires a large amount of data, as well as abundant
computation resources. While collaborative learning (e.g., federated learning)
provides a promising paradigm to harness collective data from many
participants, training large models remains a major challenge for participants
with limited resources like mobile devices. We introduce MSfusion, an effective
and efficient collaborative learning framework, tailored for training larger
models on resourceconstraint machines through model splitting. Specifically, a
double shifting model splitting scheme is designed such that in each training
round, each participant is assigned a subset of model parameters to train over
local data, and aggregates with sub-models of other peers on common parameters.
While model splitting significantly reduces the computation and communication
costs of individual participants, additional novel designs on adaptive model
overlapping and contrastive loss functions help MSfusion to maintain training
effectiveness, against model shift across participants. Extensive experiments
on image and NLP tasks illustrate significant advantages of MSfusion in
performance and efficiency for training large models, and its strong
scalability: computation cost of each participant reduces significantly as the
number of participants increases.
