  Large language models (LLMs) have shown impressive capabilities in tasks such
as machine translation, text summarization, question answering, and solving
complex mathematical problems. However, their primary training on data-rich
languages like English limits their performance in low-resource languages. This
study addresses this gap by focusing on the Indexical Shift problem in Turkish.
The Indexical Shift problem involves resolving pronouns in indexical shift
contexts, a grammatical challenge not present in high-resource languages like
English. We present the first study examining indexical shift in any language,
releasing a Turkish dataset specifically designed for this purpose. Our
Indexical Shift Dataset consists of 156 multiple-choice questions, each
annotated with necessary linguistic details, to evaluate LLMs in a few-shot
setting. We evaluate recent multilingual LLMs, including GPT-4, GPT-3.5,
Cohere-AYA, Trendyol-LLM, and Turkcell-LLM, using this dataset. Our analysis
reveals that even advanced models like GPT-4 struggle with the grammatical
nuances of indexical shift in Turkish, achieving only moderate performance.
These findings underscore the need for focused research on the grammatical
challenges posed by low-resource languages. We released the dataset and code
\href{https://anonymous.4open.science/r/indexical_shift_llm-E1B4} {here}.
