  Data $\textit{quality}$ is a crucial factor in the performance of machine
learning models, a principle that dataset distillation methods exploit by
compressing training datasets into much smaller counterparts that maintain
similar downstream performance. Understanding how and why data distillation
methods work is vital not only for improving these methods but also for
revealing fundamental characteristics of "good" training data. However, a major
challenge in achieving this goal is the observation that distillation
approaches, which rely on sophisticated but mostly disparate methods to
generate synthetic data, have little in common with each other. In this work,
we highlight a largely overlooked aspect common to most of these methods: the
use of soft (probabilistic) labels. Through a series of ablation experiments,
we study the role of soft labels in depth. Our results reveal that the main
factor explaining the performance of state-of-the-art distillation methods is
not the specific techniques used to generate synthetic data but rather the use
of soft labels. Furthermore, we demonstrate that not all soft labels are
created equal; they must contain $\textit{structured information}$ to be
beneficial. We also provide empirical scaling laws that characterize the
effectiveness of soft labels as a function of images-per-class in the distilled
dataset and establish an empirical Pareto frontier for data-efficient learning.
Combined, our findings challenge conventional wisdom in dataset distillation,
underscore the importance of soft labels in learning, and suggest new
directions for improving distillation methods. Code for all experiments is
available at https://github.com/sunnytqin/no-distillation.
