  Humans have remarkable capacity to reason abductively and hypothesize about
what lies beyond the literal content of an image. By identifying concrete
visual clues scattered throughout a scene, we almost can't help but draw
probable inferences beyond the literal scene based on our everyday experience
and knowledge about the world. For example, if we see a "20 mph" sign alongside
a road, we might assume the street sits in a residential area (rather than on a
highway), even if no houses are pictured. Can machines perform similar visual
reasoning?
  We present Sherlock, an annotated corpus of 103K images for testing machine
capacity for abductive reasoning beyond literal image contents. We adopt a
free-viewing paradigm: participants first observe and identify salient clues
within images (e.g., objects, actions) and then provide a plausible inference
about the scene, given the clue. In total, we collect 363K (clue, inference)
pairs, which form a first-of-its-kind abductive visual reasoning dataset. Using
our corpus, we test three complementary axes of abductive reasoning. We
evaluate the capacity of models to: i) retrieve relevant inferences from a
large candidate corpus; ii) localize evidence for inferences via bounding
boxes, and iii) compare plausible inferences to match human judgments on a
newly-collected diagnostic corpus of 19K Likert-scale judgments. While we find
that fine-tuning CLIP-RN50x64 with a multitask objective outperforms strong
baselines, significant headroom exists between model performance and human
agreement. Data, models, and leaderboard available at
http://visualabduction.com/
