Lune

ACL Findings2026

DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing

Hongzhi Zhang, Yuanze Hu, Tinghai Zhang, Jia Fu, Tao Wang, Junwei Jing, Zhaoxin Fan, Wei Bi, Ruiming Tang, Han Li, Guorui Zhou, Kun Gai

Abstract

The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage-where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports-remains under-evaluated due to the subjectivity of open-ended writing.

To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic "plan-then-write" workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents cites GAIA: a benchmark for General AI Assistants
  • GAIA: a benchmark for General AI Assistants cites Measuring Massive Multitask Language Understanding
  • WideSearch: Benchmarking Agentic Broad Info-Seeking cites DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
  • WideSearch: Benchmarking Agentic Broad Info-Seeking cites GAIA: a benchmark for General AI Assistants
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. GAIA: a benchmark for General AI AssistantsGrégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun et al.ICLR 2024 · 716 citations
  3. DeepResearch Bench: A Comprehensive Benchmark for Deep Research AgentsMingxuan Du, Benfeng Xu, Chiwei Zhu, Licheng Zhang et al.ICLR 2026 · 250 citations
  4. LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
  5. WideSearch: Benchmarking Agentic Broad Info-SeekingRyan Wong, Jiawei Wang, Junjie Zhao, Li Chen et al.ICLR 2026 · 66 citations

Related papers

  1. A Benchmark for Deep Information SynthesisDebjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas et al.ICLR 2026 · 1 citation
  2. WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep ResearchZijian Li, Xin Guan, Bo Zhang, Shen Huang et al.ICLR 2026 · 48 citations
  3. DeepWriter: A Multi-Agent Collaboration Framework for Information-rich Ultra-long Book WritingMing Wang, Minghao Hu, Xiuli Kang, Li He et al.AAAI 2026
  4. EpiK-Eval: Evaluation for Language Models as Epistemic ModelsGabriele Prato, Jerry Huang, Prasanna Parthasarathi, Shagun Sodhani et al.EMNLP 2023
  5. Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report RevisionBingsen Chen, Boyan Li, Ping Nie, Yuyu Zhang et al.ACL 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines