  We present a benchmark for assessing the capability of Large Language Models
(LLMs) to discern intercardinal directions between geographic locations and
apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark
specifically evaluates whether LLMs exhibit a hierarchical spatial bias similar
to humans, where judgments about individual locations' spatial relationships
are influenced by the perceived relationships of the larger groups that contain
them. To investigate this, we formulated 14 questions focusing on well-known
American cities. Seven questions were designed to challenge the LLMs with
scenarios potentially influenced by the orientation of larger geographical
units, such as states or countries, while the remaining seven targeted
locations were less susceptible to such hierarchical categorization. Among the
tested models, GPT-4 exhibited superior performance with 55 percent accuracy,
followed by GPT-3.5 at 47 percent, and Llama-2 at 45 percent. The models showed
significantly reduced accuracy on tasks with suspected hierarchical bias. For
example, GPT-4's accuracy dropped to 33 percent on these tasks, compared to 86
percent on others. However, the models identified the nearest cardinal
direction in most cases, reflecting their associative learning mechanism,
thereby embodying human-like misconceptions. We discuss avenues for improving
the spatial reasoning capabilities of LLMs.
