  Large-scale vision-language pre-trained (VLP) models are prone to hallucinate
non-existent visual objects when generating text based on visual information.
In this paper, we systematically study the object hallucination problem from
three aspects. First, we examine recent state-of-the-art VLP models, showing
that they still hallucinate frequently, and models achieving better scores on
standard metrics (e.g., CIDEr) could be more unfaithful. Second, we investigate
how different types of image encoding in VLP influence hallucination, including
region-based, grid-based, and patch-based. Surprisingly, we find that
patch-based features perform the best and smaller patch resolution yields a
non-trivial reduction in object hallucination. Third, we decouple various VLP
objectives and demonstrate that token-level image-text alignment and controlled
generation are crucial to reducing hallucination. Based on that, we propose a
simple yet effective VLP loss named ObjMLM to further mitigate object
hallucination. Results show that it reduces object hallucination by up to 17.4%
when tested on two benchmarks (COCO Caption for in-domain and NoCaps for
out-of-domain evaluation).
