  Spatial reasoning plays a vital role in both human cognition and machine
intelligence, prompting new research into language models' (LMs) capabilities
in this regard. However, existing benchmarks reveal shortcomings in evaluating
qualitative spatial reasoning (QSR). These benchmarks typically present
oversimplified scenarios or unclear natural language descriptions, hindering
effective evaluation. We present a novel benchmark for assessing QSR in LMs,
which is grounded in realistic 3D simulation data, offering a series of diverse
room layouts with various objects and their spatial relationships. This
approach provides a more detailed and context-rich narrative for spatial
reasoning evaluation, diverging from traditional, toy-task-oriented scenarios.
Our benchmark encompasses a broad spectrum of qualitative spatial
relationships, including topological, directional, and distance relations.
These are presented with different viewing points, varied granularities, and
density of relation constraints to mimic real-world complexities. A key
contribution is our logic-based consistency-checking tool, which enables the
assessment of multiple plausible solutions, aligning with real-world scenarios
where spatial relationships are often open to interpretation. Our benchmark
evaluation of advanced LMs reveals their strengths and limitations in spatial
reasoning. They face difficulties with multi-hop spatial reasoning and
interpreting a mix of different view descriptions, pointing to areas for future
improvement.
