  While there is much excitement about the potential of large multimodal models
(LMM), a comprehensive evaluation is critical to establish their true
capabilities and limitations. In support of this aim, we evaluate two
state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering
dataset sourced from an authentic online question answering community. We
conduct fine-grained analysis by generating seven types of metadata for nearly
2,000 visual questions, such as image type and the required image processing
capabilities. Our zero-shot performance analysis highlights the types of
questions that are most challenging for both models, including questions
related to "puzzling" topic, with "Identification" user intention, with "Sheet
Music" image type, or labeled as "hard" by GPT-4.
