ACL Findings2024
PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMs
Jiuzhou Han, Nigel Collier, Wray L. Buntine, Ehsan Shareghi
Abstract
Large language models (LLMs) have shown great abilities of solving various natural language tasks in different domains. Due to the training objective of LLMs and their pretraining data, LLMs are not very well equipped for tasks involving structured data generation. We propose a framework, Prompting with Iterative Verification (PiVe), to improve graphbased generative capability of LLMs. We show how a small language model could be trained to act as a verifier module for the output of an LLM (i.e., ChatGPT, GPT-4), and to iteratively improve its performance via fine-grained corrective instructions. We also show how the verifier module could apply iterative corrections offline for a more cost-effective solution to the text-to-graph generation task. Experiments on three graph-based datasets show consistent improvement gained via PiVe. Additionally, we create GenWiki-HIQ and highlight that the verifier module can be used as a data augmentation tool to help improve the quality of automatically generated parallel text-graph datasets. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction cites HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
- Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation cites HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
- Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graph cites LoRA: Low-Rank Adaptation of Large Language Models
- SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graph cites Large Language Models are Zero-Shot Reasoners
- SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graph cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Can LLMs be Good Graph Judge for Knowledge Graph Construction? cites LoRA: Low-Rank Adaptation of Large Language Models
Cited by top-tier papers11
- HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language ModelsBernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga et al.NeurIPS 2024 · 395 citations
- SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge GraphHanzhu Chen, Xu Shen, Qitan Lv, Jie Wang et al.ACL 2024 · 17 citations
- Can LLMs be Good Graph Judge for Knowledge Graph Construction?Haoyu Huang, Chong Chen, Zeang Sheng, Yang Li et al.EMNLP 2025 · 7 citations
- Scaling Knowledge Graph Construction through Synthetic Data Generation and DistillationPrafulla Kumar Choubey, Xin Su, Man Luo, XIANGYU PENG et al.ICLR 2026 · 5 citations
- AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph ConstructionHong Ting Tsang, Jiaxin Bai, Haoyu Huang, Qiao Xiao et al.ACL 2026 · 4 citations
Builds on13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
Related papers
- InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference AlignmentJianing Wang, Junda Wu, Yupeng Hou, Yao Liu et al.ACL Findings 2024 · 15 citations
- On the self-verification limitations of large language models on reasoning and planning tasksKaya Stechly, Karthik Valmeekam, Subbarao KambhampatiICLR 2025
- Towards AI-Assisted Synthesis of Verified Dafny MethodsMd Rakib Hossain Misu, Cristina V. Lopes, Iris Ma, James NobleFSE 2024 · 26 citations
- Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side VerificationHaoyang Liu, Jie Wang, Boxuan Niu, Xiongwei Han et al.ICML 2026 · 4 citations
- Can Language Models Solve Graph Problems in Natural Language?Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan et al.NeurIPS 2023 · 420 citations
