  One characteristic that makes humans superior to modern artificially
intelligent models is the ability to interpret images beyond what is visually
apparent. Consider the following two natural language search queries - (i) "a
queue of customers patiently waiting to buy ice cream" and (ii) "a queue of
tourists going to see a famous Mughal architecture in India." Interpreting
these queries requires one to reason with (i) Commonsense such as interpreting
people as customers or tourists, actions as waiting to buy or going to see; and
(ii) Fact or world knowledge associated with named visual entities, for
example, whether the store in the image sells ice cream or whether the landmark
in the image is a Mughal architecture located in India. Such reasoning goes
beyond just visual recognition. To enable both commonsense and factual
reasoning in the image search, we present a unified framework, namely Knowledge
Retrieval-Augmented Multimodal Transformer (KRAMT), that treats the named
visual entities in an image as a gateway to encyclopedic knowledge and
leverages them along with natural language query to ground relevant knowledge.
Further, KRAMT seamlessly integrates visual content and grounded knowledge to
learn alignment between images and search queries. This unified framework is
then used to perform image search requiring commonsense and factual reasoning.
The retrieval performance of KRAMT is evaluated and compared with related
approaches on a new dataset we introduce - namely COFAR. We make our code and
dataset available at https://vl2g.github.io/projects/cofar
