Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation

In recent years, large language models (LLMs) have emerged as promising candidates for graph tasks. Many studies leverage natural language to describe graphs and apply LLMs for reasoning, yet most focus narrowly on performance benchmarks without fully comparing LLMs to graph learning models or exploring their broader potential. In this work, we present a comprehensive study of LLMs on graph learning tasks, evaluating both off-theshelf and instruction-tuned models across a variety of scenarios. Beyond accuracy, we discuss data leakage concerns and computational overhead, and assess their performance under fewshot/zero-shot settings, domain transfer, structural understanding, and robustness. Our findings show that LLMs, particularly those with instruction tuning, greatly outperform traditional graph learning models in few-shot settings, exhibit strong domain transferability, and demonstrate excellent generalization and robustness. Our study highlights the broader capabilities of LLMs in graph learning and provides a foundation for future research 1 . ? ? Off-the-shelf LLMs (LLMs without parameter optimization) <Target node> <Node attribute> <1-hop neighbors> <Node attributes> <2-hop neighbors> <Node attributes> <Target node1> <Node attribute> <Target node2> <Node attribute> … … Node classification Link prediction Encode graph to text <1-hop neighbors> <Node attributes> Give you a graph language that describes a graph structure and node information from Products dataset. <Target node> <Node attribute>, known neighbor papers at hop 1: <1-hop neighbors> <Node attributes>, known neighbor papers at hop 2: <2-hop neighbors> <Node attributes> Question: please predict the most appropriate category for the Target node. Choose from the following categories: <Categories> #if original, add "Do not provide your reasoning" #if CoT, add "Let's think step by step" #if BAG, add "Let's construct a graph with the nodes and edges first" #if in-context few-shot, add 3 concrete question-answer examples. Books Based on the Products dataset, determine whether two target nodes are connected by an edge. <Target node1> <Node attribute>, known neighbor papers at hop 1: <1-hop neighbors> <Node attributes>, known neighbor papers at hop 2: <2-hop neighbors> <Node attributes> <Target node2> <Node attribute>, known neighbor papers at hop 1: <1-hop neighbors> <Node attributes>, known neighbor papers at hop 2: <2-hop neighbors> <Node attributes> Question: Are Target Node1 and Target Node2 connected? Do not provide your reasoning. Only provide "Yes" or "No" based on your inference.