  Multipanel images, commonly seen as web screenshots, posters, etc., pervade
our daily lives. These images, characterized by their composition of multiple
subfigures in distinct layouts, effectively convey information to people.
Toward building advanced multimodal AI applications, such as agents that
understand complex scenes and navigate through webpages, the skill of
multipanel visual reasoning is essential, and a comprehensive evaluation of
models in this regard is important. Therefore, we introduce Multipanel Visual
Question Answering (MultipanelVQA), a novel benchmark comprising 6,600 triplets
of questions, answers, and multipanel images that specifically challenge models
in comprehending multipanel images. Our evaluation shows that questions in the
MultipanelVQA benchmark pose significant challenges to the state-of-the-art
Multimodal Large Language Models (MLLMs) tested, even though humans can attain
approximately 99% accuracy on these questions. Distinctively, the MultipanelVQA
benchmark features synthetically generated multipanel images specifically
crafted to isolate and assess the impact of various factors, such as the
layout, on MLLMs' multipanel image comprehension abilities. As a result, in
addition to benchmarking the capabilities of MLLMs in understanding multipanel
images, we analyze various factors of the multipanel image that affect MLLMs'
performance with synthetic data and offer insights for enhancement. Code and
data are released at https://sites.google.com/view/multipanelvqa/home.
