  Today data is often scattered among billions of resource-constrained edge
devices with security and privacy constraints. Federated Learning (FL) has
emerged as a viable solution to learn a global model while keeping data
private, but the model complexity of FL is impeded by the computation resources
of edge nodes. In this work, we investigate a novel paradigm to take advantage
of a powerful server model to break through model capacity in FL. By
selectively learning from multiple teacher clients and itself, a server model
develops in-depth knowledge and transfers its knowledge back to clients in
return to boost their respective performance. Our proposed framework achieves
superior performance on both server and client models and provides several
advantages in a unified framework, including flexibility for heterogeneous
client architectures, robustness to poisoning attacks, and communication
efficiency between clients and server on various image classification tasks.
