  The use of modern Natural Language Processing (NLP) techniques has shown to
be beneficial for software engineering tasks, such as vulnerability detection
and type inference. However, training deep NLP models requires significant
computational resources. This paper explores techniques that aim at achieving
the best usage of resources and available information in these models.
  We propose a generic approach, EarlyBIRD, to build composite representations
of code from the early layers of a pre-trained transformer model. We
empirically investigate the viability of this approach on the CodeBERT model by
comparing the performance of 12 strategies for creating composite
representations with the standard practice of only using the last encoder
layer.
  Our evaluation on four datasets shows that several early layer combinations
yield better performance on defect detection, and some combinations improve
multi-class classification. More specifically, we obtain a +2 average
improvement of detection accuracy on Devign with only 3 out of 12 layers of
CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early
layers can be used to obtain better results using the same resources, as well
as to reduce resource usage during fine-tuning and inference.
