  Large Language Models (LLMs) have shown significant promise in
decision-making tasks when fine-tuned on specific applications, leveraging
their inherent common sense and reasoning abilities learned from vast amounts
of data. However, these systems are exposed to substantial safety and security
risks during the fine-tuning phase. In this work, we propose the first
comprehensive framework for Backdoor Attacks against LLM-enabled
Decision-making systems (BALD), systematically exploring how such attacks can
be introduced during the fine-tuning phase across various channels.
Specifically, we propose three attack mechanisms and corresponding backdoor
optimization methods to attack different components in the LLM-based
decision-making pipeline: word injection, scenario manipulation, and knowledge
injection. Word injection embeds trigger words directly into the query prompt.
Scenario manipulation occurs in the physical environment, where a high-level
backdoor semantic scenario triggers the attack. Knowledge injection conducts
backdoor attacks on retrieval augmented generation (RAG)-based LLM systems,
strategically injecting word triggers into poisoned knowledge while ensuring
the information remains factually accurate for stealthiness. We conduct
extensive experiments with three popular LLMs (GPT-3.5, LLaMA2, PaLM2), using
two datasets (HighwayEnv, nuScenes), and demonstrate the effectiveness and
stealthiness of our backdoor triggers and mechanisms. Finally, we critically
assess the strengths and weaknesses of our proposed approaches, highlight the
inherent vulnerabilities of LLMs in decision-making tasks, and evaluate
potential defenses to safeguard LLM-based decision making systems.
