  Multi-agent systems, powered by large language models, have shown great
abilities across various tasks due to the collaboration of expert agents, each
focusing on a specific domain. However, when agents are deployed separately,
there is a risk that malicious users may introduce malicious agents who
generate incorrect or irrelevant results that are too stealthy to be identified
by other non-specialized agents. Therefore, this paper investigates two
essential questions: (1) What is the resilience of various multi-agent system
structures (e.g., A$\rightarrow$B$\rightarrow$C,
A$\leftrightarrow$B$\leftrightarrow$C) under malicious agents, on different
downstream tasks? (2) How can we increase system resilience to defend against
malicious agents? To simulate malicious agents, we devise two methods,
AutoTransform and AutoInject, to transform any agent into a malicious one while
preserving its functional integrity. We run comprehensive experiments on four
downstream multi-agent systems tasks, namely code generation, math problems,
translation, and text evaluation. Results suggest that the "hierarchical"
multi-agent structure, i.e., A$\rightarrow$(B$\leftrightarrow$C), exhibits
superior resilience with the lowest performance drop of $23.6\%$, compared to
$46.4\%$ and $49.8\%$ of other two structures. Additionally, we show the
promise of improving multi-agent system resilience by demonstrating that two
defense methods, introducing an additional agent to review and correct messages
or mechanisms for each agent to challenge others' outputs, can enhance system
resilience. Our code and data are available at
https://github.com/CUHK-ARISE/MAS-Resilience.
