  Understanding spatial relations is essential for intelligent agents to act
and communicate in the physical world. Relative directions are spatial
relations that describe the relative positions of target objects with regard to
the intrinsic orientation of reference objects. Grounding relative directions
is more difficult than grounding absolute directions because it not only
requires a model to detect objects in the image and to identify spatial
relation based on this information, but it also needs to recognize the
orientation of objects and integrate this information into the reasoning
process. We investigate the challenging problem of grounding relative
directions with end-to-end neural networks. To this end, we provide GRiD-3D, a
novel dataset that features relative directions and complements existing visual
question answering (VQA) datasets, such as CLEVR, that involve only absolute
directions. We also provide baselines for the dataset with two established
end-to-end VQA models. Experimental evaluations show that answering questions
on relative directions is feasible when questions in the dataset simulate the
necessary subtasks for grounding relative directions. We discover that those
subtasks are learned in an order that reflects the steps of an intuitive
pipeline for processing relative directions.
