  Recognizing spatial relations and reasoning about them is essential in
multiple applications including navigation, direction giving and human-computer
interaction in general. Spatial relations between objects can either be
explicit -- expressed as spatial prepositions, or implicit -- expressed by
spatial verbs such as moving, walking, shifting, etc. Both these, but implicit
relations in particular, require significant common sense understanding. In
this paper, we introduce the task of inferring implicit and explicit spatial
relations between two entities in an image. We design a model that uses both
textual and visual information to predict the spatial relations, making use of
both positional and size information of objects and image embeddings. We
contrast our spatial model with powerful language models and show how our
modeling complements the power of these, improving prediction accuracy and
coverage and facilitates dealing with unseen subjects, objects and relations.
