  In an era marked by the increasing adoption of Large Language Models (LLMs)
for various tasks, there is a growing focus on exploring LLMs' capabilities in
handling web data, particularly graph data. Dynamic graphs, which capture
temporal network evolution patterns, are ubiquitous in real-world web data.
Evaluating LLMs' competence in understanding spatial-temporal information on
dynamic graphs is essential for their adoption in web applications, which
remains unexplored in the literature. In this paper, we bridge the gap via
proposing to evaluate LLMs' spatial-temporal understanding abilities on dynamic
graphs, to the best of our knowledge, for the first time. Specifically, we
propose the LLM4DyG benchmark, which includes nine specially designed tasks
considering the capability evaluation of LLMs from both temporal and spatial
dimensions. Then, we conduct extensive experiments to analyze the impacts of
different data generators, data statistics, prompting techniques, and LLMs on
the model performance. Finally, we propose Disentangled Spatial-Temporal
Thoughts (DST2) for LLMs on dynamic graphs to enhance LLMs' spatial-temporal
understanding abilities. Our main observations are: 1) LLMs have preliminary
spatial-temporal understanding abilities on dynamic graphs, 2) Dynamic graph
tasks show increasing difficulties for LLMs as the graph size and density
increase, while not sensitive to the time span and data generation mechanism,
3) the proposed DST2 prompting method can help to improve LLMs'
spatial-temporal understanding abilities on dynamic graphs for most tasks. The
data and codes are publicly available at Github.
