  In the context of real-world applications, leveraging large language models
(LLMs) for domain-specific tasks often faces two major challenges:
domain-specific knowledge privacy and constrained resources. To address these
issues, we propose PDSS, a privacy-preserving framework for step-by-step
distillation of LLMs. PDSS works on a server-client architecture, wherein
client transmits perturbed prompts to the server's LLM for rationale
generation. The generated rationales are then decoded by the client and used to
enrich the training of task-specific small language model(SLM) within a
multi-task learning paradigm. PDSS introduces two privacy protection
strategies: the Exponential Mechanism Strategy and the Encoder-Decoder
Strategy, balancing prompt privacy and rationale usability. Experiments
demonstrate the effectiveness of PDSS in various text generation tasks,
enabling the training of task-specific SLM with enhanced performance while
prioritizing data privacy protection.
