Precise Information Control in Long-Form Text Generation

A central challenge in language models (LMs) is faithfulness hallucination: the generation of information unsubstantiated by input context. To study this problem, we propose Precise Information Control (PIC), a new task formulation that requires models to generate long-form outputs grounded in a provided set of short self-contained statements, without adding any unsupported ones. PIC includes a full setting that tests a model's ability to include exactly all input claims, and a partial setting that requires the model to selectively incorporate only relevant claims. We present PIC-Bench, a benchmark of eight long-form generation tasks (e.g., summarization, biography generation) adapted to the PIC setting, where LMs are supplied with well-formed, verifiable input claims. Our evaluation of a range of open and proprietary LMs on PIC-Bench reveals that, surprisingly, state-of-the-art LMs still hallucinate against user-provided input in over 70% of generations. To alleviate this lack of faithfulness, we introduce a post-training framework that uses a weakly supervised preference data construction method to train an 8B PIC-LM with stronger PIC ability-improving from 69.1% to 91.0% F 1 in the full PIC setting. When integrated into end-to-end factual generation pipelines, PIC-LM improves exact match recall by 17.1% on ambiguous QA with retrieval, and factual precision by 30.5% on a birthplace fact-checking task, underscoring the potential of precisely grounded generation.

Task Name PIC Type N C Example Instruction I EntityBiosPIC Full 183 50.5 Generate a factual biography about Suthida. PopBios-PPIC Full 111 20.1 Give me a biography on Erwin Schrödinger, the scientist who discovered Quantum Mechanics, Schrödinger's Cat Thought Experiment. PopBios-CFPIC Full 111 20.1 Give me a biography on Oscar Wilde, the scientist who discovered Quantum Mechanics, Schrödinger's Cat Thought Experiment. ELI5PIC Full 146 12.5 Answer the following question(s): why it's common to have 87-octane gasoline in the US but it's almost always 95-octane in Europe? AskHistoriansPIC Full 158 19.2 In the original Star Wars: A New Hope, Obi-Wan Kenobi instructs R2-D2 to connect to the Imperial network to gain access to the whole system. Did the concept of an interconnected vast computer network exist in 1977? ExpertQA PIC Full 152 13.5 Answer the question(s): What's the difference between modern and contemporary architecture? FACTSPIC Partial 150 63.5 Explain the benefits of using mobile technology to improve healthcare management in both hi-income and low-income countries. Context p XSUMPIC Partial 200 30.6 Summarize the following text in around 20-25 words. Context p