  This paper proposes a pipeline for quantitatively evaluating interactive LLMs
such as ChatGPT using publicly available dataset. We carry out an extensive
technical evaluation of ChatGPT using Big-Vul covering five different common
software vulnerability tasks. We evaluate the multitask and multilingual
aspects of ChatGPT based on this dataset. We found that the existing
state-of-the-art methods are generally superior to ChatGPT in software
vulnerability detection. Although ChatGPT improves accuracy when providing
context information, it still has limitations in accurately predicting severity
ratings for certain CWE types. In addition, ChatGPT demonstrates some ability
in locating vulnerabilities for certain CWE types, but its performance varies
among different CWE types. ChatGPT exhibits limited vulnerability repair
capabilities in both providing and not providing context information. Finally,
ChatGPT shows uneven performance in generating CVE descriptions for various CWE
types, with limited accuracy in detailed information. Overall, though ChatGPT
performs well in some aspects, it still needs improvement in understanding the
subtle differences in code vulnerabilities and the ability to describe
vulnerabilities in order to fully realize its potential. Our evaluation
framework provides valuable insights for further enhancing ChatGPT' s software
vulnerability handling capabilities.
