  The emergence of large-scale pre-trained language models has revolutionized
various AI research domains. Transformers-based Large Language Models (LLMs)
have gradually replaced CNNs and RNNs to unify fields of computer vision and
natural language processing. Compared with independent data samples such as
images, videos or texts, graphs usually contain rich structural and relational
information. Meanwhile, language, especially natural language, being one of the
most expressive mediums, excels in describing complex structures. However,
existing work on incorporating graph problems into the generative language
modeling framework remains very limited. Considering the rising prominence of
LLMs, it becomes essential to explore whether LLMs can also replace GNNs as the
foundation model for graphs. In this paper, we propose InstructGLM
(Instruction-finetuned Graph Language Model) with highly scalable prompts based
on natural language instructions. We use natural language to describe
multi-scale geometric structure of the graph and then instruction finetune an
LLM to perform graph tasks, which enables Generative Graph Learning. Our method
surpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets,
underscoring its effectiveness and sheds light on generative LLMs as new
foundation model for graph machine learning. Our code is open-sourced at
https://github.com/agiresearch/InstructGLM.
