  Large Language Models (LLMs) such as ChatGPT and Llama-2 have become
prevalent in real-world applications, exhibiting impressive text generation
performance. LLMs are fundamentally developed from a scenario where the input
data remains static and lacks a clear structure. To behave interactively over
time, LLM-based chat systems must integrate additional contextual information
(i.e., chat history) into their inputs, following a pre-defined structure. This
paper identifies how such integration can expose LLMs to misleading context
from untrusted sources and fail to differentiate between system and user
inputs, allowing users to inject context. We present a systematic methodology
for conducting context injection attacks aimed at eliciting disallowed
responses by introducing fabricated context. This could lead to illegal
actions, inappropriate content, or technology misuse. Our context fabrication
strategies, acceptance elicitation and word anonymization, effectively create
misleading contexts that can be structured with attacker-customized prompt
templates, achieving injection through malicious user messages. Comprehensive
evaluations on real-world LLMs such as ChatGPT and Llama-2 confirm the efficacy
of the proposed attack with success rates reaching 97%. We also discuss
potential countermeasures that can be adopted for attack detection and
developing more secure models. Our findings provide insights into the
challenges associated with the real-world deployment of LLMs for interactive
and structured data scenarios.
