  Federated Split Learning (FSL) is a promising distributed learning paradigm
in practice, which gathers the strengths of both Federated Learning (FL) and
Split Learning (SL) paradigms, to ensure model privacy while diminishing the
resource overhead of each client, especially on large transformer models in a
resource-constrained environment, e.g., Internet of Things (IoT). However,
almost all works merely investigate the performance with simple neural network
models in FSL. Despite the minor efforts focusing on incorporating Vision
Transformers (ViT) as model architectures, they train ViT from scratch, thereby
leading to enormous training overhead in each device with limited resources.
Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as
the initial model, coined FedV, to accelerate the training process and improve
model robustness. Furthermore, we propose FedVZ to hinder the gradient
inversion attack, especially having the capability compatible with black-box
scenarios, where the gradient information is unavailable. Concretely, FedVZ
approximates the server gradient by utilizing a zeroth-order (ZO) optimization,
which replaces the backward propagation with just one forward process.
Empirically, we are the first to provide a systematic evaluation of FSL methods
with PITs in real-world datasets, different partial device participations, and
heterogeneous data splits. Our experiments verify the effectiveness of our
algorithms.
