  To reduce the size of recommendation models, there have been many studies on
compressing recommendation models using knowledge distillation. In this paper,
we decompose recommendation models into three layers, i.e., the input layer,
the intermediate layer, and the output layer, and address deficiencies layer by
layer. First, previous methods focus only on two layers, neglecting the input
layer. Second, in the intermediate layer, existing methods ignore the
inconsistency of user preferences induced by the projectors. Third, in the
output layer, existing methods use only hard labels rather than soft labels
from the teacher. To address these deficiencies, we propose
\textbf{M}ulti-layer \textbf{K}nowledge \textbf{D}istillation (MKD), which
consists of three components: 1) Distillation with Neighbor-based Knowledge
(NKD) utilizes the teacher's knowledge about entities with similar
characteristics in the input layer to enable the student to learn robust
representations. 2) Distillation with Consistent Preference (CPD) reduces the
inconsistency of user preferences caused by projectors in the intermediate
layer by two regularization terms. 3) Distillation with Soft Labels (SLD)
constructs soft labels in the output layer by considering the predictions of
both the teacher and the student. Our extensive experiments show that MKD even
outperforms the teacher with one-tenth of the model size.
