ACL Findings2026
Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xinyun Liu, Yulia Tsvetkov
Abstract
Previous research has sought to enhance the graph reasoning capabilities of LLMs by supervised fine-tuning on synthetic graph data.While these led to specialized LLMs better at solving graph algorithm problems, we don't need LLMs for shortest path: we need generalization from synthetic graph data to realworld tasks with implicit graph structures.In this work, we propose to unlock generalizable learning of graph with post-training alignment with synthetic data.We first design solutionbased and process-based rewards for synthetic graph problems: instead of rigid memorizing response patterns in direct fine-tuning, we posit that post-training alignment would help LLMs grasp the essentials underlying graph reasoning and alleviate overfitting on synthetic data.We employ post-training alignment algorithms such as GRPO and DPO, aligning both off-theshelf LLMs and LLMs fine-tuned on synthetic graph data.We then compare them against existing settings on both in-domain synthetic tasks and out-of-domain real-world tasks with implicit graph structures such as multi-hop QA, structured planning, and more.Extensive experiments demonstrate that our post-training alignment recipe leads to statistically significant improvement on 5 datasets, with an average gain of 12.9% over baseline settings.Further analysis reveals that process-based rewards consistently outperform solution-based rewards on synthetic data but not on real-world tasks, and compositionality and explainable intermediate steps remains a critical challenge even after post-training alignment.1 and Bryan Hooi.2024b.G-retriever: Retrievalaugmented generation for textual graph understanding and question answering.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale cites Language Models are Few-Shot Learners
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Language Models are Few-Shot Learners
Builds on43
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
Related papers
- Can LLM Graph Reasoning Generalize beyond Pattern Memorization?Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan et al.EMNLP Findings 2024 · 13 citations
- Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise SupervisionGe Chang, Jinbo Su, Jiacheng Liu, Pengfei Yang et al.ACL 2026 · 1 citation
- GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement LearningChuanyue Yu, Kuo Zhao, Yuhan Li, Heng Chang et al.WWW 2026 · 8 citations
- G1: Teaching LLMs to Reason on Graphs with Reinforcement LearningXiaojun Guo, Ang Li, Yifei Wang, Stefanie Jegelka et al.NeurIPS 2025 · 16 citations
- InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference AlignmentJianing Wang, Junda Wu, Yupeng Hou, Yao Liu et al.ACL Findings 2024 · 15 citations
