  Federated learning (FL) is a promising paradigm to enable privacy-preserving
deep learning from distributed data. Most previous works are based on federated
average (FedAvg), which, however, faces several critical issues, including a
high communication overhead and the difficulty in dealing with heterogeneous
model architectures. Federated Distillation (FD) is a recently proposed
alternative to enable communication-efficient and robust FL, which achieves
orders of magnitude reduction of the communication overhead compared with
FedAvg and is flexible to handle heterogeneous models at the clients. However,
so far there is no unified algorithmic framework or theoretical analysis for
FD-based methods. In this paper, we first present a generic meta-algorithm for
FD and investigate the influence of key parameters through empirical
experiments. Then, we verify the empirical observations theoretically. Based on
the empirical results and theory, we propose a communication-efficient FD
algorithm with active data sampling to improve the model performance and reduce
the communication overhead. Empirical simulations on benchmark datasets will
demonstrate that our proposed algorithm effectively and significantly reduces
the communication overhead while achieving a satisfactory performance.
