  Adopting human and large language models (LLM) as judges (a.k.a human- and
LLM-as-a-judge) for evaluating the performance of LLMs has recently gained
attention. Nonetheless, this approach concurrently introduces potential biases
from human and LLMs, questioning the reliability of the evaluation results. In
this paper, we propose a novel framework that is free from referencing
groundtruth annotations for investigating Misinformation Oversight Bias, Gender
Bias, Authority Bias and Beauty Bias on LLM and human judges. We curate a
dataset referring to the revised Bloom's Taxonomy and conduct thousands of
evaluations. Results show that human and LLM judges are vulnerable to
perturbations to various degrees, and that even the cutting-edge judges possess
considerable biases. We further exploit these biases to conduct attacks on LLM
judges. We hope that our work can notify the community of the bias and
vulnerability of human- and LLM-as-a-judge, as well as the urgency of
developing robust evaluation systems.
