  Large Language Models (LLMs) have showcased exceptional capabilities in
various domains, attracting significant interest from both academia and
industry. Despite their impressive performance, the substantial size and
computational demands of LLMs pose considerable challenges for practical
deployment, particularly in environments with limited resources. The endeavor
to compress language models while maintaining their accuracy has become a focal
point of research. Among the various methods, knowledge distillation has
emerged as an effective technique to enhance inference speed without greatly
compromising performance. This paper presents a thorough survey from three
aspects: method, evaluation, and application, exploring knowledge distillation
techniques tailored specifically for LLMs. Specifically, we divide the methods
into white-box KD and black-box KD to better illustrate their differences.
Furthermore, we also explored the evaluation tasks and distillation effects
between different distillation methods, and proposed directions for future
research. Through in-depth understanding of the latest advancements and
practical applications, this survey provides valuable resources for
researchers, paving the way for sustained progress in this field.
