  Despite remarkable advancements in mitigating hallucinations in large
language models (LLMs) by retrieval augmentation, it remains challenging to
measure the reliability of LLMs using static question-answering (QA) data.
Specifically, given the potential of data contamination (e.g., leading to
memorization), good static benchmark performance does not ensure that model can
reliably use the provided evidence for responding, which is essential to avoid
hallucination when the required knowledge is new or private. Inspired by
adversarial machine learning, we investigate the feasibility of automatically
perturbing existing static one for dynamic evaluation. Specifically, this paper
presents ReEval, an LLM-based framework using prompt chaining to perturb the
original evidence for generating new test cases for evaluating the LLMs'
reliability in using new evidence for answering.
  We implement ReEval using ChatGPT and evaluate the resulting variants of two
popular open-domain QA datasets on a collection of LLMs under various prompting
settings. Our generated data is human-readable and useful to trigger
hallucination in LLM. Accurate models on static data are observed to produce
unsupported answers from the perturbed evidence, with pronounced accuracy drops
across LLMs including GPT-4. We find that our adversarial examples are
transferable across all considered LLMs. The examples generated by a small
model can be used to evaluate a much larger model, making our approach
cost-effective.
