  Reducing the `$\textit{hallucination}$' problem of Large Language Models
(LLMs) is crucial for their wide applications. A comprehensive and fine-grained
measurement of the hallucination is the first key step for the governance of
this issue but is under-explored in the community. Thus, we present
$\textbf{ANAH}$, a bilingual dataset that offers $\textbf{AN}$alytical
$\textbf{A}$nnotation of $\textbf{H}$allucinations in LLMs within Generative
Question Answering. Each answer sentence in our dataset undergoes rigorous
annotation, involving the retrieval of a reference fragment, the judgment of
the hallucination type, and the correction of hallucinated content. ANAH
consists of ~12k sentence-level annotations for ~4.3k LLM responses covering
over 700 topics, constructed by a human-in-the-loop pipeline. Thanks to the
fine granularity of the hallucination annotations, we can quantitatively
confirm that the hallucinations of LLMs progressively accumulate in the answer
and use ANAH to train and evaluate hallucination annotators. We conduct
extensive experiments on studying generative and discriminative annotators and
show that, although current open-source LLMs have difficulties in fine-grained
hallucination annotation, the generative annotator trained with ANAH can
surpass all open-source LLMs and GPT-3.5, obtain performance competitive with
GPT-4, and exhibits better generalization ability on unseen questions.
