  Heterogeneity of data distributed across clients limits the performance of
global models trained through federated learning, especially in the settings
with highly imbalanced class distributions of local datasets. In recent years,
personalized federated learning (pFL) has emerged as a potential solution to
the challenges presented by heterogeneous data. However, existing pFL methods
typically enhance performance of local models at the expense of the global
model's accuracy. We propose FedHKD (Federated Hyper-Knowledge Distillation), a
novel FL algorithm in which clients rely on knowledge distillation (KD) to
train local models. In particular, each client extracts and sends to the server
the means of local data representations and the corresponding soft predictions
-- information that we refer to as ``hyper-knowledge". The server aggregates
this information and broadcasts it to the clients in support of local training.
Notably, unlike other KD-based pFL methods, FedHKD does not rely on a public
dataset nor it deploys a generative model at the server. We analyze convergence
of FedHKD and conduct extensive experiments on visual datasets in a variety of
scenarios, demonstrating that FedHKD provides significant improvement in both
personalized as well as global model performance compared to state-of-the-art
FL methods designed for heterogeneous data settings.
