  Large Language Models (LLMs) have demonstrated exceptional proficiency in
instruction-following, becoming increasingly crucial across various
applications. However, this capability brings with it the risk of prompt
injection attacks, where attackers inject instructions into LLMs' input to
elicit undesirable actions or content. Understanding the robustness of LLMs
against such attacks is vital for their safe implementation. In this work, we
establish a benchmark to evaluate the robustness of instruction-following LLMs
against prompt injection attacks. Our objective is to determine the extent to
which LLMs can be influenced by injected instructions and their ability to
differentiate between these injected and original target instructions. Through
extensive experiments with leading instruction-following LLMs, we uncover
significant vulnerabilities in their robustness to such attacks. Our results
indicate that some models are overly tuned to follow any embedded instructions
in the prompt, overly focusing on the latter parts of the prompt without fully
grasping the entire context. By contrast, models with a better grasp of the
context and instruction-following capabilities will potentially be more
susceptible to compromise by injected instructions. This underscores the need
to shift the focus from merely enhancing LLMs' instruction-following
capabilities to improving their overall comprehension of prompts and
discernment of instructions that are appropriate to follow. We hope our
in-depth analysis offers insights into the underlying causes of these
vulnerabilities, aiding in the development of future solutions. Code and data
are available at
https://github.com/Leezekun/instruction-following-robustness-eval
