  Temporal reasoning (TR) is a critical component of artificial intelligence,
encompassing understanding and processing temporal information and
relationships between events. To discover and study the TR ability in Large
Language Models (LLMs), various datasets have been constructed in different
ways for evaluating various aspects of TR ability. Our work proposes a novel
approach to design and develop a pipeline for constructing datasets to evaluate
the TR ability of LLMs by leveraging random directed graph generation, LTL
formula, and the NuSMV model checker. Based on the pipeline, we have also
constructed a dataset as a benchmark, namely LTLBench, consisting of 2,000 TR
challenges and evaluated six LLMs with it. Furthermore, we have conducted
additional experiments to discover the impact of increasing the number of
events and formula operators on the complexity of TR problems and the
performance of LLMs. We have demonstrated that although LLMs exhibit some
promise in handling TR challenges, they still struggle with complex TR. We
expect this work can offer insights into TR ability in LLMs while also
providing a valuable tool for future TR evaluations.
