  Although large pre-trained models of code have delivered significant
advancements in various code processing tasks, there is an impediment to the
wide and fluent adoption of these powerful models in software developers' daily
workflow: these large models consume hundreds of megabytes of memory and run
slowly on personal devices, which causes problems in model deployment and
greatly degrades the user experience.
  It motivates us to propose Compressor, a novel approach that can compress the
pre-trained models of code into extremely small models with negligible
performance sacrifice. Our proposed method formulates the design of tiny models
as simplifying the pre-trained model architecture: searching for a
significantly smaller model that follows an architectural design similar to the
original pre-trained model. Compressor proposes a genetic algorithm (GA)-based
strategy to guide the simplification process. Prior studies found that a model
with higher computational cost tends to be more powerful. Inspired by this
insight, the GA algorithm is designed to maximize a model's Giga floating-point
operations (GFLOPs), an indicator of the model computational cost, to satisfy
the constraint of the target model size. Then, we use the knowledge
distillation technique to train the small model: unlabelled data is fed into
the large model and the outputs are used as labels to train the small model. We
evaluate Compressor with two state-of-the-art pre-trained models, i.e.,
CodeBERT and GraphCodeBERT, on two important tasks, i.e., vulnerability
prediction and clone detection. We use our method to compress pre-trained
models to a size (3 MB), which is 160$\times$ smaller than the original size.
The results show that compressed CodeBERT and GraphCodeBERT are 4.31$\times$
and 4.15$\times$ faster than the original model at inference, respectively.
More importantly, ...
