  Mainstream poisoning attacks on large language models (LLMs) typically set a
fixed trigger in the input instance and specific responses for triggered
queries. However, the fixed trigger setting (e.g., unusual words) may be easily
detected by human detection, limiting the effectiveness and practicality in
real-world scenarios. To enhance the stealthiness of the trigger, we present a
poisoning attack against LLMs that is triggered by a generation/output
condition-token limitation, which is a commonly adopted strategy by users for
reducing costs. The poisoned model performs normally for output without token
limitation, while becomes harmful for output with limited tokens. To achieve
this objective, we introduce BrieFool, an efficient attack framework. It
leverages the characteristics of generation limitation by efficient instruction
sampling and poisoning data generation, thereby influencing the behavior of
LLMs under target conditions. Our experiments demonstrate that BrieFool is
effective across safety domains and knowledge domains. For instance, with only
20 generated poisoning examples against GPT-3.5-turbo, BrieFool achieves a 100%
Attack Success Rate (ASR) and a 9.28/10 average Harmfulness Score (HS) under
token limitation conditions while maintaining the benign performance.
