  Pre-trained language models have demonstrated powerful capabilities in the
field of natural language processing (NLP). Recently, code pre-trained model
(PTM), which draw from the experiences of the NLP field, have also achieved
state-of-the-art results in many software engineering (SE) downstream tasks.
These code PTMs take into account the differences between programming languages
and natural languages during pre-training and make adjustments to pre-training
tasks and input data. However, researchers in the SE community still inherit
habits from the NLP field when using these code PTMs to generate embeddings for
SE downstream classification tasks, such as generating semantic embeddings for
code snippets through special tokens and inputting code and text information in
the same way as pre-training the PTMs. In this paper, we empirically study five
different PTMs (i.e. CodeBERT, CodeT5, PLBART, CodeGPT and CodeGen) with three
different architectures (i.e. encoder-only, decoder-only and encoder-decoder)
on four SE downstream classification tasks (i.e. code vulnerability detection,
code clone detection, just-in-time defect prediction and function docstring
mismatch detection) with respect to the two aforementioned aspects. Our
experimental results indicate that (1) regardless of the architecture of the
code PTMs used, embeddings obtained through special tokens do not sufficiently
aggregate the semantic information of the entire code snippet; (2) the quality
of code embeddings obtained by combing code data and text data in the same way
as pre-training the PTMs is poor and cannot guarantee richer semantic
information; (3) using the method that aggregates the vector representations of
all code tokens, the decoder-only PTMs can obtain code embeddings with
semantics as rich as or even better quality than those obtained from the
encoder-only and encoder-decoder PTMs.
