  Learning representations of spatial references in natural language is a key
challenge in tasks like autonomous navigation and robotic manipulation. Recent
work has investigated various neural architectures for learning multi-modal
representations for spatial concepts. However, the lack of explicit reasoning
over entities makes such approaches vulnerable to noise in input text or state
observations. In this paper, we develop effective models for understanding
spatial references in text that are robust and interpretable, without
sacrificing performance. We design a text-conditioned \textit{relation network}
whose parameters are dynamically computed with a cross-modal attention module
to capture fine-grained spatial relations between entities. This design choice
provides interpretability of learned intermediate outputs. Experiments across
three tasks demonstrate that our model achieves superior performance, with a
17\% improvement in predicting goal locations and a 15\% improvement in
robustness compared to state-of-the-art systems.
