  Large Language Models (LLMs) represent formidable tools for sequence
modeling, boasting an innate capacity for general pattern recognition.
Nevertheless, their broader spatial reasoning capabilities, especially applied
to numerical trajectory data, remain insufficiently explored. In this paper, we
investigate the out-of-the-box performance of ChatGPT-3.5, ChatGPT-4 and Llama
2 7B models when confronted with 3D robotic trajectory data from the CALVIN
baseline and associated tasks, including 2D directional and shape labeling.
Additionally, we introduce a novel prefix-based prompting mechanism, which
yields a 33% improvement on the 3D trajectory data and an increase of up to 10%
on SpartQA tasks over zero-shot prompting (with gains for other prompting types
as well). The experimentation with 3D trajectory data offers an intriguing
glimpse into the manner in which LLMs engage with numerical and spatial
information, thus laying a solid foundation for the identification of target
areas for future enhancements.
