Efficient Real-time Refinement of Language Model Text Generation
Abstract
Large language models (LLMs) have shown remarkable performance across a wide range of natural language tasks. However, a critical challenge remains in that they sometimes generate factually incorrect answers. To address this, while many previous work has focused on identifying errors in their generation and further refining them, they are slow in deployment since they are designed to verify the response from LLMs only after their entire generation (from the first to last tokens) is done. Further, we observe that once LLMs generate incorrect tokens early on, there is a higher likelihood that subsequent tokens will also be factually incorrect. To this end, in this work, we propose Streaming-VR (Streaming Verification and Refinement), a novel approach designed to enhance the efficiency of verification and refinement of LLM outputs. Specifically, the proposed Streaming-VR enables on-the-fly verification and correction of tokens as they are being generated, similar to a streaming process, ensuring that each subset of tokens is checked and refined in realtime by another LLM as the LLM constructs its response. Through comprehensive evaluations on multiple datasets, we demonstrate that our approach not only enhances the factual accuracy of LLMs, but also offers a more efficient solution compared to prior refinement methods. Q: Who wrote harry styles sign of the times lyrics? Sign of the Times is the debut single by English singer Harry Styles from his self-titled debut studio album.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Self-Refine: Iterative Refinement with Self-Feedback
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval Augmented Language Model Pre-Training
Builds on23
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
Related papers
- Knowledge-Augmented Language Model VerificationJinheon Baek, Soyeong Jeong, Minki Kang, Jong C. Park et al.EMNLP 2023 · 12 citations
- RAC: Efficient LLM Factuality Correction with Retrieval AugmentationChangmao Li, Jeffrey FlaniganEMNLP Findings 2025
- A Stitch in Time Saves Nine: Proactive Self-Refinement for Language ModelsJinyi Han, Xinyi Wang, Haiquan Zhao, tingyun li et al.ICLR 2026 · 2 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
- Towards Verifiable Text Generation with Evolving Memory and Self-ReflectionHao Sun, Hengyi Cai, Bo Wang, Yingyan Hou et al.EMNLP 2024 · 6 citations
