  Large Language Models (LLMs) have demonstrated remarkable success in tasks
like the Winograd Schema Challenge (WSC), showcasing advanced textual
common-sense reasoning. However, applying this reasoning to multimodal domains,
where understanding text and images together is essential, remains a
substantial challenge. To address this, we introduce WinoVis, a novel dataset
specifically designed to probe text-to-image models on pronoun disambiguation
within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion
Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel
evaluation framework that isolates the models' ability in pronoun
disambiguation from other visual processing challenges. Evaluation of
successive model versions reveals that, despite incremental advancements,
Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally
surpassing random guessing. Further error analysis identifies important areas
for future research aimed at advancing text-to-image models in their ability to
interpret and interact with the complex visual world.
