Lune

ACL Findings2026

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xinyun Liu, Yulia Tsvetkov

Abstract

Previous research has sought to enhance the graph reasoning capabilities of LLMs by supervised fine-tuning on synthetic graph data.While these led to specialized LLMs better at solving graph algorithm problems, we don't need LLMs for shortest path: we need generalization from synthetic graph data to realworld tasks with implicit graph structures.In this work, we propose to unlock generalizable learning of graph with post-training alignment with synthetic data.We first design solutionbased and process-based rewards for synthetic graph problems: instead of rigid memorizing response patterns in direct fine-tuning, we posit that post-training alignment would help LLMs grasp the essentials underlying graph reasoning and alleviate overfitting on synthetic data.We employ post-training alignment algorithms such as GRPO and DPO, aligning both off-theshelf LLMs and LLMs fine-tuned on synthetic graph data.We then compare them against existing settings on both in-domain synthetic tasks and out-of-domain real-world tasks with implicit graph structures such as multi-hop QA, structured planning, and more.Extensive experiments demonstrate that our post-training alignment recipe leads to statistically significant improvement on 5 datasets, with an average gain of 12.9% over baseline settings.Further analysis reveals that process-based rewards consistently outperform solution-based rewards on synthetic data but not on real-world tasks, and compositionality and explainable intermediate steps remains a critical challenge even after post-training alignment.1 and Bryan Hooi.2024b.G-retriever: Retrievalaugmented generation for textual graph understanding and question answering.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DAPO: An Open-Source LLM Reinforcement Learning System at Scale cites Language Models are Few-Shot Learners
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on43

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  4. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  5. SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations

Related papers

  1. Can LLM Graph Reasoning Generalize beyond Pattern Memorization?Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan et al.EMNLP Findings 2024 · 13 citations
  2. Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise SupervisionGe Chang, Jinbo Su, Jiacheng Liu, Pengfei Yang et al.ACL 2026 · 1 citation
  3. GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement LearningChuanyue Yu, Kuo Zhao, Yuhan Li, Heng Chang et al.WWW 2026 · 8 citations
  4. G1: Teaching LLMs to Reason on Graphs with Reinforcement LearningXiaojun Guo, Ang Li, Yifei Wang, Stefanie Jegelka et al.NeurIPS 2025 · 16 citations
  5. InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference AlignmentJianing Wang, Junda Wu, Yupeng Hou, Yao Liu et al.ACL Findings 2024 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines