  Pre-trained code models have emerged as crucial tools in various code
intelligence tasks. However, their effectiveness depends on the quality of the
pre-training dataset, particularly the human reference comments, which serve as
a bridge between the programming language and natural language. One significant
challenge is that such comments can become inconsistent with the corresponding
code as the software evolves. This discrepancy can lead to suboptimal training
of the models, decreasing their performances. LLMs have demonstrated superior
capabilities in generating high-quality code comments. In light of that, we try
to tackle the quality issue of the dataset by harnessing the power of LLMs.
Specifically, we raise the question: Can we rebuild the pre-training dataset by
substituting the original comments with LLM-generated ones for more effective
pre-trained code models? To answer the question, we first conduct a
comprehensive evaluation to compare ChatGPT-generated comments with human
reference comments. As existing reference-based metrics treat the reference
comments as gold standards, we introduce two auxiliary tasks as novel
reference-free metrics to assess the quality of comments, i.e., code-comment
inconsistency detection and code search. Experimental results show that
ChatGPT-generated comments demonstrate superior semantic consistency with the
code compared to human references, indicating the potential of utilizing
ChatGPT to enhance the quality of the pre-training dataset. We rebuilt the
widely used dataset, CodeSearchNet, with ChatGPT-generated comments. Subsequent
experiments involve re-pre-training the CodeT5 with our refined
dataset.Evaluation results on four generation tasks and one understanding code
intelligence tasks show that the model pre-trained by ChatGPT-enhanced data
outperforms its counterpart on code summarization, code generation, and code
translation tasks.
