  The rapid evolution of deep learning and large language models has led to an
exponential growth in the demand for training data, prompting the development
of Dataset Distillation methods to address the challenges of managing large
datasets. Among these, Matching Training Trajectories (MTT) has been a
prominent approach, which replicates the training trajectory of an expert
network on real data with a synthetic dataset. However, our investigation found
that this method suffers from three significant limitations: 1. Instability of
expert trajectory generated by Stochastic Gradient Descent (SGD); 2. Low
convergence speed of the distillation process; 3. High storage consumption of
the expert trajectory. To address these issues, we offer a new perspective on
understanding the essence of Dataset Distillation and MTT through a simple
transformation of the objective function, and introduce a novel method called
Matching Convexified Trajectory (MCT), which aims to provide better guidance
for the student trajectory. MCT leverages insights from the linearized dynamics
of Neural Tangent Kernel methods to create a convex combination of expert
trajectories, guiding the student network to converge rapidly and stably. This
trajectory is not only easier to store, but also enables a continuous sampling
strategy during distillation, ensuring thorough learning and fitting of the
entire expert trajectory. Comprehensive experiments across three public
datasets validate the superiority of MCT over traditional MTT methods.
