  With the growing popularity of Large Language Models (LLMs) in software
engineers' daily practices, it is important to ensure that the code generated
by these tools is not only functionally correct but also free of
vulnerabilities. Although LLMs can help developers to be more productive, prior
empirical studies have shown that LLMs can generate insecure code. There are
two contributing factors to the insecure code generation. First, existing
datasets used to evaluate LLMs do not adequately represent genuine software
engineering tasks sensitive to security. Instead, they are often based on
competitive programming challenges or classroom-type coding tasks. In
real-world applications, the code produced is integrated into larger codebases,
introducing potential security risks. Second, existing evaluation metrics
primarily focus on the functional correctness of the generated code while
ignoring security considerations. Therefore, in this paper, we described SALLM,
a framework to benchmark LLMs' abilities to generate secure code
systematically. This framework has three major components: a novel dataset of
security-centric Python prompts, configurable assessment techniques to evaluate
the generated code, and novel metrics to evaluate the models' performance from
the perspective of secure code generation.
