  Dataset Distillation (DD), a newly emerging field, aims at generating much
smaller but efficient synthetic training datasets from large ones. Existing DD
methods based on gradient matching achieve leading performance; however, they
are extremely computationally intensive as they require continuously optimizing
a dataset among thousands of randomly initialized models. In this paper, we
assume that training the synthetic data with diverse models leads to better
generalization performance. Thus we propose two model augmentation techniques,
i.e. using early-stage models and parameter perturbation to learn an
informative synthetic set with significantly reduced training cost. Extensive
experiments demonstrate that our method achieves up to 20x speedup and
comparable performance on par with state-of-the-art methods.
