  This paper presents the results of finetuning large language models (LLMs)
for the task of detecting vulnerabilities in source code. We leverage
WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and
adapt it for vulnerability detection through further finetuning. To accelerate
training, we modify WizardCoder's training procedure, also we investigate
optimal training regimes. For the imbalanced dataset with many more negative
examples than positive, we also explore different techniques to improve
classification performance. The finetuned WizardCoder model achieves
improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability
datasets over CodeBERT-like model, demonstrating the effectiveness of adapting
pretrained LLMs for vulnerability detection in source code. The key
contributions are finetuning the state-of-the-art code LLM, WizardCoder,
increasing its training speed without the performance harm, optimizing the
training procedure and regimes, handling class imbalance, and improving
performance on difficult vulnerability detection datasets. This demonstrates
the potential for transfer learning by finetuning large pretrained language
models for specialized source code analysis tasks.
