  Knowledge distillation (KD) is known as a promising solution to compress
large language models (LLMs) via transferring their knowledge to smaller
models. During this process, white-box KD methods usually minimize the distance
between the output distributions of the two models so that more knowledge can
be transferred. However, in the current white-box KD framework, the output
distributions are from the respective output spaces of the two models, using
their own prediction heads. We argue that the space discrepancy will lead to
low similarity between the teacher model and the student model on both
representation and distribution levels. Furthermore, this discrepancy also
hinders the KD process between models with different vocabularies, which is
common for current LLMs. To address these issues, we propose a dual-space
knowledge distillation (DSKD) framework that unifies the output spaces of the
two models for KD. On the basis of DSKD, we further develop a cross-model
attention mechanism, which can automatically align the representations of the
two models with different vocabularies. Thus, our framework is not only
compatible with various distance functions for KD (e.g., KL divergence) like
the current framework, but also supports KD between any two LLMs regardless of
their vocabularies. Experiments on task-agnostic instruction-following
benchmarks show that DSKD significantly outperforms the current white-box KD
framework with various distance functions, and also surpasses existing KD
methods for LLMs with different vocabularies.
