  Data heterogeneity presents significant challenges for federated learning
(FL). Recently, dataset distillation techniques have been introduced, and
performed at the client level, to attempt to mitigate some of these challenges.
In this paper, we propose a highly efficient FL dataset distillation framework
on the server side, significantly reducing both the computational and
communication demands on local devices while enhancing the clients' privacy.
Unlike previous strategies that perform dataset distillation on local devices
and upload synthetic data to the server, our technique enables the server to
leverage prior knowledge from pre-trained deep generative models to synthesize
essential data representations from a heterogeneous model architecture. This
process allows local devices to train smaller surrogate models while enabling
the training of a larger global model on the server, effectively minimizing
resource utilization. We substantiate our claim with a theoretical analysis,
demonstrating the asymptotic resemblance of the process to the hypothetical
ideal of completely centralized training on a heterogeneous dataset. Empirical
evidence from our comprehensive experiments indicates our method's superiority,
delivering an accuracy enhancement of up to 40% over non-dataset-distillation
techniques in highly heterogeneous FL contexts, and surpassing existing
dataset-distillation methods by 18%. In addition to the high accuracy, our
framework converges faster than the baselines because rather than the server
trains on several sets of heterogeneous data distributions, it trains on a
multi-modal distribution. Our code is available at
https://github.com/FedDG23/FedDG-main.git
