On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
Zhitao He, Haolin Yang, Rui Min, Zeyu Qin, Yi Fung
Abstract
Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output (i.e., the inconsistency in length and content across multiple generations). This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: benchmarking, probing, and mitigation. We first propose the VOlatility in Long-form Text Benchmark (VOLTBench), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose SELB (Structural Enforcement via Logits Boosting), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148% and reduces the length volatility by 69%, while maintaining high generation quality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention cites Efficient Streaming Language Models with Attention Sinks
- ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education cites MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQL
Cited by top-tier papers2
- MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQLHaolin Yang, Jipeng Zhang, Zhitao He, Alexander Zhou et al.ICML 2026 · 12 citations
- ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic EducationZhitao He, Haolin Yang, Zeyu Qin, Yi FungICML 2026 · 4 citations
Builds on16
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionJingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo et al.ACL 2025 · 334 citations
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMsXiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen et al.NeurIPS 2025 · 63 citations
- MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQLHaolin Yang, Jipeng Zhang, Zhitao He, Alexander Zhou et al.ICML 2026 · 12 citations
- LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement LearningYuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee et al.ICLR 2026 · 9 citations
Related papers
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMsYuhao Wu, Ming Shan Hee, Zhiqiang Hu, Roy Ka-Wei LeeICLR 2025
- PIC: Unlocking Long-Form Text Generation Capabilities of Large Language Models via Position ID CompressionHaoran Que, Wenge RongACL 2025 · 2 citations
- ReFF: Reinforcing Format Faithfulness in Language Models Across Varied TasksJiashu Yao, Heyan Huang, Zeming Liu, Haoyu Wen et al.AAAI 2025 · 1 citation
- LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
- LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language ModelsWei Zhang, Lintong Du, yuanhe zhang, Zhenhong Zhou et al.ICML 2026
