  Federated learning (FL) is vulnerable to heterogeneously distributed data,
since a common global model in FL may not adapt to the heterogeneous data
distribution of each user. To counter this issue, personalized FL (PFL) was
proposed to produce dedicated local models for each individual user. However,
PFL is far from its maturity, because existing PFL solutions either demonstrate
unsatisfactory generalization towards different model architectures or cost
enormous extra computation and memory. In this work, we propose federated
learning with personalized sparse mask (FedSpa), a novel PFL scheme that
employs personalized sparse masks to customize sparse local models on the edge.
Instead of training an intact (or dense) PFL model, FedSpa only maintains a
fixed number of active parameters throughout training (aka sparse-to-sparse
training), which enables users' models to achieve personalization with cheap
communication, computation, and memory cost. We theoretically show that the
iterates obtained by FedSpa converge to the local minimizer of the formulated
SPFL problem at rate of $\mathcal{O}(\frac{1}{\sqrt{T}})$. Comprehensive
experiments demonstrate that FedSpa significantly saves communication and
computation costs, while simultaneously achieves higher model accuracy and
faster convergence speed against several state-of-the-art PFL methods.
