  Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train
high-performance student models using only the pre-trained teacher network
without original training data. Most of the existing DFKD methods rely heavily
on additional generation modules to synthesize the substitution data resulting
in high computational costs and ignoring the massive amounts of easily
accessible, low-cost, unlabeled open-world data. Meanwhile, existing methods
ignore the domain shift issue between the substitution data and the original
data, resulting in knowledge from teachers not always trustworthy and
structured knowledge from data becoming a crucial supplement. To tackle the
issue, we propose a novel Open-world Data Sampling Distillation (ODSD) method
for the DFKD task without the redundant generation process. First, we try to
sample open-world data close to the original data's distribution by an adaptive
sampling module and introduce a low-noise representation to alleviate the
domain shift issue. Then, we build structured relationships of multiple data
examples to exploit data knowledge through the student model itself and the
teacher's structured representation. Extensive experiments on CIFAR-10,
CIFAR-100, NYUv2, and ImageNet show that our ODSD method achieves
state-of-the-art performance with lower FLOPs and parameters. Especially, we
improve 1.50\%-9.59\% accuracy on the ImageNet dataset and avoid training the
separate generator for each class.
