  As deep learning models and datasets rapidly scale up, network training is
extremely time-consuming and resource-costly. Instead of training on the entire
dataset, learning with a small synthetic dataset becomes an efficient solution.
Extensive research has been explored in the direction of dataset condensation,
among which gradient matching achieves state-of-the-art performance. The
gradient matching method directly targets the training dynamics by matching the
gradient when training on the original and synthetic datasets. However, there
are limited deep investigations into the principle and effectiveness of this
method. In this work, we delve into the gradient matching method from a
comprehensive perspective and answer the critical questions of what, how, and
where to match. We propose to match the multi-level gradients to involve both
intra-class and inter-class gradient information. We demonstrate that the
distance function should focus on the angle, considering the magnitude
simultaneously to delay the overfitting. An overfitting-aware adaptive learning
step strategy is also proposed to trim unnecessary optimization steps for
algorithmic efficiency improvement. Ablation and comparison experiments
demonstrate that our proposed methodology shows superior accuracy, efficiency,
and generalization compared to prior work.
