  Federated Learning (FL) stands to gain significant advantages from
collaboratively training capacity-heterogeneous models, enabling the
utilization of private data and computing power from low-capacity devices.
However, the focus on personalizing capacity-heterogeneous models based on
client-specific data has been limited, resulting in suboptimal local model
utility, particularly for low-capacity clients. The heterogeneity in both data
and device capacity poses two key challenges for model personalization: 1)
accurately retaining necessary knowledge embedded within reduced submodels for
each client, and 2) effectively sharing knowledge through aggregating
size-varying parameters. To this end, we introduce Pa3dFL, a novel framework
designed to enhance local model performance by decoupling and selectively
sharing knowledge among capacity-heterogeneous models. First, we decompose each
layer of the model into general and personal parameters. Then, we maintain
uniform sizes for the general parameters across clients and aggregate them
through direct averaging. Subsequently, we employ a hyper-network to generate
size-varying personal parameters for clients using learnable embeddings.
Finally, we facilitate the implicit aggregation of personal parameters by
aggregating client embeddings through a self-attention module. We conducted
extensive experiments on three datasets to evaluate the effectiveness of
Pa3dFL. Our findings indicate that Pa3dFL consistently outperforms baseline
methods across various heterogeneity settings. Moreover, Pa3dFL demonstrates
competitive communication and computation efficiency compared to baseline
approaches, highlighting its practicality and adaptability in adverse system
conditions.
