  Data-free knowledge distillation (DFKD) conducts knowledge distillation via
eliminating the dependence of original training data, and has recently achieved
impressive results in accelerating pre-trained language models. At the heart of
DFKD is to reconstruct a synthetic dataset by inverting the parameters of the
uncompressed model. Prior DFKD approaches, however, have largely relied on
hand-crafted priors of the target data distribution for the reconstruction,
which can be inevitably biased and often incompetent to capture the intrinsic
distributions. To address this problem, we propose a prompt-based method,
termed as PromptDFD, that allows us to take advantage of learned language
priors, which effectively harmonizes the synthetic sentences to be semantically
and grammatically correct. Specifically, PromptDFD leverages a pre-trained
generative model to provide language priors and introduces a reinforced topic
prompter to control data synthesis, making the generated samples thematically
relevant and semantically plausible, and thus friendly to downstream tasks. As
shown in our experiments, the proposed method substantially improves the
synthesis quality and achieves considerable improvements on distillation
performance. In some cases, PromptDFD even gives rise to results on par with
those from the data-driven knowledge distillation with access to the original
training data.
