  Federated Learning (FL) requires frequent exchange of model parameters, which
leads to long communication delay, especially when the network environments of
clients vary greatly. Moreover, the parameter server needs to wait for the
slowest client (i.e., straggler, which may have the largest model size, lowest
computing capability or worst network condition) to upload parameters, which
may significantly degrade the communication efficiency. Commonly-used client
selection methods such as partial client selection would lead to the waste of
computing resources and weaken the generalization of the global model. To
tackle this problem, along a different line, in this paper, we advocate the
approach of model parameter dropout instead of client selection, and
accordingly propose a novel framework of Federated learning scheme with
Differential parameter Dropout (FedDD). FedDD consists of two key modules:
dropout rate allocation and uploaded parameter selection, which will optimize
the model parameter uploading ratios tailored to different clients'
heterogeneous conditions and also select the proper set of important model
parameters for uploading subject to clients' dropout rate constraints.
Specifically, the dropout rate allocation is formulated as a convex
optimization problem, taking system heterogeneity, data heterogeneity, and
model heterogeneity among clients into consideration. The uploaded parameter
selection strategy prioritizes on eliciting important parameters for uploading
to speedup convergence. Furthermore, we theoretically analyze the convergence
of the proposed FedDD scheme. Extensive performance evaluations demonstrate
that the proposed FedDD scheme can achieve outstanding performances in both
communication efficiency and model convergence, and also possesses a strong
generalization capability to data of rare classes.
