  This paper introduces UnSeenTimeQA, a novel time-sensitive question-answering
(TSQA) benchmark that diverges from traditional TSQA benchmarks by avoiding
factual and web-searchable queries. We present a series of time-sensitive event
scenarios decoupled from real-world factual information. It requires large
language models (LLMs) to engage in genuine temporal reasoning, disassociating
from the knowledge acquired during the pre-training phase. Our evaluation of
six open-source LLMs (ranging from 2B to 70B in size) and three closed-source
LLMs reveal that the questions from the UnSeenTimeQA present substantial
challenges. This indicates the models' difficulties in handling complex
temporal reasoning scenarios. Additionally, we present several analyses
shedding light on the models' performance in answering time-sensitive
questions.
