  Imagine observing someone scratching their arm; to understand why, additional
context would be necessary. However, spotting a mosquito nearby would
immediately offer a likely explanation for the person's discomfort, thereby
alleviating the need for further information. This example illustrates how
subtle visual cues can challenge our cognitive skills and demonstrates the
complexity of interpreting visual scenarios. To study these skills, we present
Visual Riddles, a benchmark aimed to test vision and language models on visual
riddles requiring commonsense and world knowledge. The benchmark comprises 400
visual riddles, each featuring a unique image created by a variety of
text-to-image models, question, ground-truth answer, textual hint, and
attribution. Human evaluation reveals that existing models lag significantly
behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
to make assessment scalable. These findings underscore the potential of Visual
Riddles as a valuable resource for enhancing vision and language models'
capabilities in interpreting complex visual scenarios.
