  Instruction-tuned Code Large Language Models (Code LLMs) are increasingly
utilized as AI coding assistants and integrated into various applications.
However, the cybersecurity vulnerabilities and implications arising from the
widespread integration of these models are not yet fully understood due to
limited research in this domain. To bridge this gap, this paper presents
EvilInstructCoder, a framework specifically designed to assess the
cybersecurity vulnerabilities of instruction-tuned Code LLMs to adversarial
attacks. EvilInstructCoder introduces the Adversarial Code Injection Engine to
automatically generate malicious code snippets and inject them into benign code
to poison instruction tuning datasets. It incorporates practical threat models
to reflect real-world adversaries with varying capabilities and evaluates the
exploitability of instruction-tuned Code LLMs under these diverse adversarial
attack scenarios. Through the use of EvilInstructCoder, we conduct a
comprehensive investigation into the exploitability of instruction tuning for
coding tasks using three state-of-the-art Code LLM models: CodeLlama,
DeepSeek-Coder, and StarCoder2, under various adversarial attack scenarios. Our
experimental results reveal a significant vulnerability in these models,
demonstrating that adversaries can manipulate the models to generate malicious
payloads within benign code contexts in response to natural language
instructions. For instance, under the backdoor attack setting, by poisoning
only 81 samples (0.5\% of the entire instruction dataset), we achieve Attack
Success Rate at 1 (ASR@1) scores ranging from 76\% to 86\% for different model
families. Our study sheds light on the critical cybersecurity vulnerabilities
posed by instruction-tuned Code LLMs and emphasizes the urgent necessity for
robust defense mechanisms to mitigate the identified vulnerabilities.
