  Dataset distillation has emerged as a strategy to overcome the hurdles
associated with large datasets by learning a compact set of synthetic data that
retains essential information from the original dataset. While distilled data
can be used to train high performing models, little is understood about how the
information is stored. In this study, we posit and answer three questions about
the behavior, representativeness, and point-wise information content of
distilled data. We reveal distilled data cannot serve as a substitute for real
data during training outside the standard evaluation setting for dataset
distillation. Additionally, the distillation process retains high task
performance by compressing information related to the early training dynamics
of real models. Finally, we provide an framework for interpreting distilled
data and reveal that individual distilled data points contain meaningful
semantic information. This investigation sheds light on the intricate nature of
distilled data, providing a better understanding on how they can be effectively
utilized.
