  Recently, innovative model aggregation methods based on knowledge
distillation (KD) have been proposed for federated learning (FL). These methods
not only improved the robustness of model aggregation over heterogeneous
learning environment, but also allowed training heterogeneous models on client
devices. However, the scalability of existing methods is not satisfactory,
because the training cost on the server increases with the number of clients,
which limits their application in large scale systems. Furthermore, the
ensemble of existing methods is built from a set of client models initialized
from the same checkpoint, causing low diversity. In this paper, we propose a
scalable and diversity-enhanced federated distillation scheme, FedSDD, which
decouples the training complexity from the number of clients to enhance the
scalability, and builds the ensemble from a set of aggregated models with
enhanced diversity. In particular, the teacher model in FedSDD is an ensemble
built by a small group of aggregated (global) models, instead of all client
models, such that the computation cost will not scale with the number of
clients. Furthermore, to enhance diversity, FedSDD only performs KD to enhance
one of the global models, i.e., the \textit{main global model}, which improves
the performance of both the ensemble and the main global model. While
partitioning client model into more groups allow building an ensemble with more
aggregated models, the convergence of individual aggregated models will be slow
down. We introduce the temporal ensembling which leverage the issues, and
provide significant improvement with the heterogeneous settings. Experiment
results show that FedSDD outperforms other FL methods, including FedAvg and
FedDF, on the benchmark datasets.
