Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generation

Despite the significant progress of large language models (LLMs) in various tasks, they often produce factual errors due to their limited internal knowledge. Retrieval-Augmented Generation (RAG), which enhances LLMs with external knowledge sources, offers a promising solution. However, these methods can be misled by irrelevant paragraphs in retrieved documents. Due to the inherent uncertainty in LLM generation, inputting the entire document may introduce off-topic information, causing the model to deviate from the central topic and affecting the relevance of the generated content. To address these issues, we propose the Retrieve-Plan-Generation (RPG) framework. RPG generates plan tokens to guide subsequent generation in the plan stage. In the answer stage, the model selects relevant fine-grained paragraphs based on the plan and uses them for further answer generation. This plan-answer process is repeated iteratively until completion, enhancing generation relevance by focusing on specific topics. To implement this framework efficiently, we utilize a simple but effective multi-task prompt-tuning method, enabling the existing LLMs to handle both planning and answering. We comprehensively compare RPG with baselines across 5 knowledge-intensive generation tasks, demonstrating the effectiveness of our approach. 1 * Corresponding author. † Equal Contribution. 1 Our code and models will be publicly available at https://github.com/haruhi-sudo/RPG . Question: How do jellyfish function without brains or nervous systems? Retrieved Documents: Title: Jellyfish, Simple yet Mysterious Ocean Creatures. Jellyfish lack a brain but possess a rudimentary nervous system and neural cells capable of sensing external stimuli such as light and touch[...] They likely coordinate these functions through a nerve ring [...] Jellyfish are invertebrates [...] Some species have remarkably brief lifespans, lasting merely hours. Yet their vitality is remarkably resilient[...] Most strikingly, jellyfish possess an astounding capacity for regeneration [...] Normal RAG: Despite their seemingly lifeless appearance, jellyfish are among the most wondrous creatures on Earth! Though lacking a brain, jellyfish possess a rudimentary nervous system and neural[...] Their bodies also exhibit a peculiar self-healing ability, rapidly regenerating even when cut into pieces and reassembled. Furthermore, jellyfish can bioluminescent, illuminating the dark [...] Plan Before Generate: [Plan: Body composition and structure of jellyfish] <p>Jellyfish…Their bodies are almost entirely made up of gelatinous hydrogel tissue, with a water content of more than 98%...</p>Jellyfish are indeed very simple creatures, almost more than 98% of which are made up of water. [Plan: Nervous system of jellyfish] <p>Jellyfish barely have nervous systems…They have primitive nerve cells that help them</p>However, although jellyfish do not have brains, they do have a rough nervous system that can sense external stimuli such as light and touch[...] [Plan: Nerve ring of jellyfish] <p>..</p>Jellyfish coordinate through nerve rings [...]