  Do the Spratly Islands belong to China, the Philippines, or Vietnam? A
pretrained large language model (LLM) may answer differently if asked in the
languages of each claimant country: Chinese, Tagalog, or Vietnamese. This
contrasts with a multilingual human, who would likely answer consistently. In
this paper, we show that LLMs recall certain geographical knowledge
inconsistently when queried in different languages -- a phenomenon we term
geopolitical bias. As a targeted case study, we consider territorial disputes,
an inherently controversial and multilingual task. We introduce BorderLines, a
dataset of territorial disputes which covers 251 territories, each associated
with a set of multiple-choice questions in the languages of each claimant
country (49 languages in total). We also propose a suite of evaluation metrics
to precisely quantify bias and consistency in responses across different
languages. We then evaluate various multilingual LLMs on our dataset and
metrics to probe their internal knowledge and use the proposed metrics to
discover numerous inconsistencies in how these models respond in different
languages. Finally, we explore several prompt modification strategies, aiming
to either amplify or mitigate geopolitical bias, which highlights how brittle
LLMs are and how they tailor their responses depending on cues from the
interaction context. Our code and data are available at
https://github.com/manestay/borderlines
