  The rapid adoption of large language models (LLMs) in multi-agent systems has
highlighted their impressive capabilities in various applications, such as
collaborative problem-solving and autonomous negotiation. However, the security
implications of these LLM-based multi-agent systems have not been thoroughly
investigated, particularly concerning the spread of manipulated knowledge. In
this paper, we investigate this critical issue by constructing a detailed
threat model and a comprehensive simulation environment that mirrors real-world
multi-agent deployments in a trusted platform. Subsequently, we propose a novel
two-stage attack method involving Persuasiveness Injection and Manipulated
Knowledge Injection to systematically explore the potential for manipulated
knowledge (i.e., counterfactual and toxic knowledge) spread without explicit
prompt manipulation.
  Our method leverages the inherent vulnerabilities of LLMs in handling world
knowledge, which can be exploited by attackers to unconsciously spread
fabricated information. Through extensive experiments, we demonstrate that our
attack method can successfully induce LLM-based agents to spread both
counterfactual and toxic knowledge without degrading their foundational
capabilities during agent communication. Furthermore, we show that these
manipulations can persist through popular retrieval-augmented generation
frameworks, where several benign agents store and retrieve manipulated chat
histories for future interactions. This persistence indicates that even after
the interaction has ended, the benign agents may continue to be influenced by
manipulated knowledge. Our findings reveal significant security risks in
LLM-based multi-agent systems, emphasizing the imperative need for robust
defenses against manipulated knowledge spread, such as introducing ``guardian''
agents and advanced fact-checking tools.
