  Large language models (LLMs) encode vast amounts of world knowledge acquired
via training on large web-scale datasets crawled from the internet. However,
these datasets typically exhibit a geographical bias towards English-speaking
Western countries. This results in LLMs producing biased or hallucinated
responses to queries that require answers localized to other geographical
regions. In this work, we introduce a new benchmark named LoFTI (Localization
and Factuality Transfer to Indian Locales) that can be used to evaluate an
LLM's localization and factual text transfer capabilities. LoFTI consists of
factual statements about entities in source and target locations; the source
locations are spread across the globe and the target locations are all within
India with varying degrees of hyperlocality (country, states, cities). The
entities span a wide variety of categories. We use LoFTI to evaluate Mixtral,
GPT-4 and two other Mixtral-based approaches well-suited to the task of
localized factual transfer. We demonstrate that LoFTI is a high-quality
evaluation benchmark and all the models, including GPT-4, produce skewed
results across varying levels of hyperlocality.
