  While GPT-4V(ision) impressively models both visual and textual information
simultaneously, it's hallucination behavior has not been systematically
assessed. To bridge this gap, we introduce a new benchmark, namely, the Bias
and Interference Challenges in Visual Language Models (Bingo). This benchmark
is designed to evaluate and shed light on the two common types of
hallucinations in visual language models: bias and interference. Here, bias
refers to the model's tendency to hallucinate certain types of responses,
possibly due to imbalance in its training data. Interference pertains to
scenarios where the judgment of GPT-4V(ision) can be disrupted due to how the
text prompt is phrased or how the input image is presented. We identify a
notable regional bias, whereby GPT-4V(ision) is better at interpreting Western
images or images with English writing compared to images from other countries
or containing text in other languages. Moreover, GPT-4V(ision) is vulnerable to
leading questions and is often confused when interpreting multiple images
together. Popular mitigation approaches, such as self-correction and
chain-of-thought reasoning, are not effective in resolving these challenges. We
also identified similar biases and interference vulnerabilities with LLaVA and
Bard. Our results characterize the hallucination challenges in GPT-4V(ision)
and state-of-the-art visual-language models, and highlight the need for new
solutions. The Bingo benchmark is available at https://github.com/gzcch/Bingo.
