  Large language models (LLMs) have brought significant advancements to code
generation and code repair, benefiting both novice and experienced developers.
However, their training using unsanitized data from open-source repositories,
like GitHub, raises the risk of inadvertently propagating security
vulnerabilities. Despite numerous studies investigating the safety of code
LLMs, there remains a gap in comprehensively addressing their security
features. In this work, we aim to present a comprehensive study aimed at
precisely evaluating and enhancing the security aspects of code LLMs. To
support our research, we introduce CodeSecEval, a meticulously curated dataset
designed to address 44 critical vulnerability types with 180 distinct samples.
CodeSecEval serves as the foundation for the automatic evaluation of code
models in two crucial tasks: code generation and code repair, with a strong
emphasis on security. Our experimental results reveal that current models
frequently overlook security issues during both code generation and repair
processes, resulting in the creation of vulnerable code. In response, we
propose different strategies that leverage vulnerability-aware information and
insecure code explanations to mitigate these security vulnerabilities.
Furthermore, our findings highlight that certain vulnerability types
particularly challenge model performance, influencing their effectiveness in
real-world applications. Based on these findings, we believe our study will
have a positive impact on the software engineering community, inspiring the
development of improved methods for training and utilizing LLMs, thereby
leading to safer and more trustworthy model deployment.
