  Large Language Models (LLMs) are powerful zero-shot assessors used in
real-world situations such as assessing written exams and benchmarking systems.
Despite these critical applications, no existing work has analyzed the
vulnerability of judge-LLMs to adversarial manipulation. This work presents the
first study on the adversarial robustness of assessment LLMs, where we
demonstrate that short universal adversarial phrases can be concatenated to
deceive judge LLMs to predict inflated scores. Since adversaries may not know
or have access to the judge-LLMs, we propose a simple surrogate attack where a
surrogate model is first attacked, and the learned attack phrase then
transferred to unknown judge-LLMs. We propose a practical algorithm to
determine the short universal attack phrases and demonstrate that when
transferred to unseen models, scores can be drastically inflated such that
irrespective of the assessed text, maximum scores are predicted. It is found
that judge-LLMs are significantly more susceptible to these adversarial attacks
when used for absolute scoring, as opposed to comparative assessment. Our
findings raise concerns on the reliability of LLM-as-a-judge methods, and
emphasize the importance of addressing vulnerabilities in LLM assessment
methods before deployment in high-stakes real-world scenarios.
