  Previous studies have proved that cross-lingual knowledge distillation can
significantly improve the performance of pre-trained models for cross-lingual
similarity matching tasks. However, the student model needs to be large in this
operation. Otherwise, its performance will drop sharply, thus making it
impractical to be deployed to memory-limited devices. To address this issue, we
delve into cross-lingual knowledge distillation and propose a multi-stage
distillation framework for constructing a small-size but high-performance
cross-lingual model. In our framework, contrastive learning, bottleneck, and
parameter recurrent strategies are combined to prevent performance from being
compromised during the compression process. The experimental results
demonstrate that our method can compress the size of XLM-R and MiniLM by more
than 50\%, while the performance is only reduced by about 1%.
