  Recent advancements in dataset distillation have demonstrated the significant
benefits of employing soft labels generated by pre-trained teacher models. In
this paper, we introduce a novel perspective by emphasizing the full
utilization of labels. We first conduct a comprehensive comparison of various
loss functions for soft label utilization in dataset distillation, revealing
that the model trained on the synthetic dataset exhibits high sensitivity to
the choice of loss function for soft label utilization. This finding highlights
the necessity of a universal loss function for training models on synthetic
datasets. Building on these insights, we introduce an extremely simple yet
surprisingly effective plug-and-play approach, GIFT, which encompasses soft
label refinement and a cosine similarity-based loss function to efficiently
leverage full label information. Extensive experiments demonstrate that GIFT
consistently enhances the state-of-the-art dataset distillation methods across
various scales datasets without incurring additional computational costs. For
instance, on ImageNet-1K with IPC = 10, GIFT improves the SOTA method RDED by
3.9% and 1.8% on ConvNet and ResNet-18, respectively. Code:
https://github.com/LINs-lab/GIFT.
