  Warning: This paper contains examples of stereotypes and biases. Large
Language Models (LLMs) exhibit considerable social biases, and various studies
have tried to evaluate and mitigate these biases accurately. Previous studies
use downstream tasks as prompts to examine the degree of social biases for
evaluation and mitigation. While LLMs' output highly depends on prompts,
previous studies evaluating and mitigating bias have often relied on a limited
variety of prompts. In this paper, we investigate the sensitivity of LLMs when
changing prompt variations (task instruction and prompt, few-shot examples,
debias-prompt) by analyzing task performance and social bias of LLMs. Our
experimental results reveal that LLMs are highly sensitive to prompts to the
extent that the ranking of LLMs fluctuates when comparing models for task
performance and social bias. Additionally, we show that LLMs have tradeoffs
between performance and social bias caused by the prompts. Less bias from
prompt setting may result in reduced performance. Moreover, the ambiguity of
instances is one of the reasons for this sensitivity to prompts in advanced
LLMs, leading to various outputs. We recommend using diverse prompts, as in
this study, to compare the effects of prompts on social bias in LLMs.
