  Artificial agents today can answer factual questions. But they fall short on
questions that require common sense reasoning. Perhaps this is because most
existing common sense databases rely on text to learn and represent knowledge.
But much of common sense knowledge is unwritten - partly because it tends not
to be interesting enough to talk about, and partly because some common sense is
unnatural to articulate in text. While unwritten, it is not unseen. In this
paper we leverage semantic common sense knowledge learned from images - i.e.
visual common sense - in two textual tasks: fill-in-the-blank and visual
paraphrasing. We propose to "imagine" the scene behind the text, and leverage
visual cues from the "imagined" scenes in addition to textual cues while
answering these questions. We imagine the scenes as a visual abstraction. Our
approach outperforms a strong text-only baseline on these tasks. Our proposed
tasks can serve as benchmarks to quantitatively evaluate progress in solving
tasks that go "beyond recognition". Our code and datasets are publicly
available.
