Lune

EMNLP Findings2024

Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models

Jiaming Li, Lei Zhang, Yunshui Li, Ziqiang Liu, Yuelin Bai, Run Luo, Longze Chen, Min Yang

Abstract

The instruction-following ability of large language models enables humans to interact with AI agents in a natural way. However, when required to generate responses of a specific length, large language models often struggle to meet users' needs due to their inherent difficulty in accurately perceiving numerical constraints. To explore the ability of large language models to control the length of generated responses, we propose the Target Length Generation Task (TLG) and design two metrics, Precise Match (PM) and Flexible Match (FM) to evaluate the model's performance in adhering to specified response lengths. Furthermore, we introduce a novel, model-agnostic approach called RULER, which employs Meta Length Tokens (MLTs) to enhance the instruction-following ability of large language models under length-constrained instructions. Specifically, RULER equips LLMs with the ability to generate responses of a specified length based on length constraints within the instructions. Moreover, RULER can automatically generate appropriate MLT when length constraints are not explicitly provided, demonstrating excellent versatility and generalization. Comprehensive experiments show the effectiveness of RULER across different LLMs on Target Length Generation Task, e.g., at All Level 27.97 average gain on PM, 29.57 average gain on FM. In addition, we conduct extensive ablation experiments to further substantiate the efficacy and generalization of RULER. Our code and data is available at https://github.com/Geaming2002/Ruler.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models cites Length Controlled Generation for Black-box LLMs
  • LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models cites Measuring Massive Multitask Language Understanding
  • LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models cites Language Models are Few-Shot Learners
  • LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models cites From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen
  • Length Controlled Generation for Black-box LLMs cites Language Models are Few-Shot Learners
  • Length Controlled Generation for Black-box LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGenPeiwen Yuan, Chuyi Tan, Shaoxiong Feng, Yiwei Li et al.ACL 2025 · 2 citations
  2. LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language ModelsWei Zhang, Lintong Du, yuanhe zhang, Zhenhong Zhou et al.ICML 2026
  3. Length Controlled Generation for Black-box LLMsYuxuan Gu, Wenjie Wang, Xiaocheng Feng, Weihong Zhong et al.ACL 2025

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations

Related papers

  1. Prompt-Based Length Controlled Generation with Multiple Control TypesRenlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang et al.ACL Findings 2024 · 4 citations
  2. LongGenBench: Benchmarking Long-Form Generation in Long Context LLMsYuhao Wu, Ming Shan Hee, Zhiqiang Hu, Roy Ka-Wei LeeICLR 2025
  3. Evaluating Large Language Models at Evaluating Instruction FollowingZhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng et al.ICLR 2024 · 299 citations
  4. Following Length Constraints in InstructionsWeizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho et al.EMNLP 2025 · 1 citation
  5. Context Length Alone Hurts LLM Performance Despite Perfect RetrievalYufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali et al.EMNLP Findings 2025 · 38 citations
Dusk over the sea between two cliffs drawn in fine vertical lines