  The heterogeneity of hardware and data is a well-known and studied problem in
the community of Federated Learning (FL) as running under heterogeneous
settings. Recently, custom-size client models trained with Knowledge
Distillation (KD) has emerged as a viable strategy for tackling the
heterogeneity challenge. However, previous efforts in this direction are aimed
at client model tuning rather than their impact onto the knowledge aggregation
of the global model. Despite performance of global models being the primary
objective of FL systems, under heterogeneous settings client models have
received more attention. Here, we provide more insights into how the chosen
approach for training custom client models has an impact on the global model,
which is essential for any FL application. We show the global model can fully
leverage the strength of KD with heterogeneous data. Driven by empirical
observations, we further propose a new approach that combines KD and Learning
without Forgetting (LwoF) to produce improved personalised models. We bring
heterogeneous FL on pair with the mighty FedAvg of homogeneous FL, in realistic
deployment scenarios with dropping clients.
