  Relevance evaluation of a query and a passage is essential in Information
Retrieval (IR). Recently, numerous studies have been conducted on tasks related
to relevance judgment using Large Language Models (LLMs) such as GPT-4,
demonstrating significant improvements. However, the efficacy of LLMs is
considerably influenced by the design of the prompt. The purpose of this paper
is to identify which specific terms in prompts positively or negatively impact
relevance evaluation with LLMs. We employed two types of prompts: those used in
previous research and generated automatically by LLMs. By comparing the
performance of these prompts in both few-shot and zero-shot settings, we
analyze the influence of specific terms in the prompts. We have observed two
main findings from our study. First, we discovered that prompts using the term
answerlead to more effective relevance evaluations than those using relevant.
This indicates that a more direct approach, focusing on answering the query,
tends to enhance performance. Second, we noted the importance of appropriately
balancing the scope of relevance. While the term relevant can extend the scope
too broadly, resulting in less precise evaluations, an optimal balance in
defining relevance is crucial for accurate assessments. The inclusion of
few-shot examples helps in more precisely defining this balance. By providing
clearer contexts for the term relevance, few-shot examples contribute to refine
relevance criteria. In conclusion, our study highlights the significance of
carefully selecting terms in prompts for relevance evaluation with LLMs.
