  Large vision-language models (LVLMs) have shown remarkable abilities in
understanding visual information with human languages. However, LVLMs still
suffer from object hallucination, which is the problem of generating
descriptions that include objects that do not actually exist in the images.
This can negatively impact many vision-language tasks, such as visual
summarization and reasoning. To address this issue, we propose a simple yet
powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify
object hallucination in LVLMs by reconstructing less hallucinatory
descriptions. LURE is grounded in a rigorous statistical analysis of the key
factors underlying object hallucination, including co-occurrence (the frequent
appearance of certain objects alongside others in images), uncertainty (objects
with higher uncertainty during LVLM decoding), and object position
(hallucination often appears in the later part of the generated text). LURE can
also be seamlessly integrated with any LVLMs. We evaluate LURE on six
open-source LVLMs, achieving a 23% improvement in general object hallucination
evaluation metrics over the previous best approach. In both GPT and human
evaluations, LURE consistently ranks at the top. Our data and code are
available at https://github.com/YiyangZhou/LURE.
