EMNLP Findings2024
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints
Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, Nanyun Peng
摘要
Instruction following is a key capability for LLMs. However, recent studies have shown that LLMs often struggle with instructions containing multiple constraints (e.g. a request to create a social media post "in a funny tone" with "no hashtag"). Despite this, most evaluations focus solely on synthetic data. To address this, we introduce REALINSTRUCT, the first benchmark designed to evaluate LLMs' ability to follow real-world multi-constrained instructions by leveraging queries real users asked AI assistants. We also investigate modelbased evaluation as a cost-effective alternative to human annotation for this task. Our findings reveal that even the proprietary GPT-4 model fails to meet at least one constraint on over 21% of instructions, highlighting the limitations of state-of-the-art models. To address the performance gap between open-source and proprietary models, we propose the DECOM-POSE, CRITIQUE, AND REFINE (DECRIM) self-correction pipeline, which enhances LLMs' ability to follow constraints. DECRIM works by decomposing the original instruction into a list of constraints and using a Critic model to decide when and where the LLM's response needs refinement. Our results show that DECRIM improves Mistral's performance by 7.3% on RE-ALINSTRUCT and 8.0% on IFEval even with weak feedback. Moreover, we demonstrate that with strong feedback, open-source LLMs with DECRIM can outperform GPT-4 on both benchmarks.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 8 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- LoRA: Low-Rank Adaptation of Large Language Models 引用了 Language Models are Few-Shot Learners
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Measuring Massive Multitask Language Understanding 引用了 Language Models are Few-Shot Learners
- Revisiting the Reliability of Language Models in Instruction-Following 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- Safe4U: Identifying Unsound Safe Encapsulations of Unsafe Calls in Rust using LLMs 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
引用它的顶会 Paper8
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language ModelsYulei Qin, Gang Li, Zongyi Li, Zihan Xu 等NeurIPS 2025 · 17 次引用
- RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint DataZhengkang Guo, Wenhao Liu, Mingchen Xie, Jingwen Xu 等ICLR 2026 · 13 次引用
- Revisiting the Reliability of Language Models in Instruction-FollowingJianshuo Dong, Yutong Zhang, Liu Yan, Zhenyu Zhong 等ACL 2026 · 3 次引用
- Safe4U: Identifying Unsound Safe Encapsulations of Unsafe Calls in Rust using LLMsHuan Li, Bei Wang, Xing Hu, Xin XiaISSTA 2025 · 1 次引用
- Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?Momoka Furuhashi, Kouta Nakayama, Takashi Kodama, Saku SugawaraEMNLP 2025 · 1 次引用
它引用的 Paper46
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma 等NeurIPS 2022 · 22,562 次引用
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger 等ICLR 2020 · 8,443 次引用
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou 等ICLR 2021 · 7,905 次引用
相关 Paper
- FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language ModelsYuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong 等ACL 2024 · 10 次引用
- ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in InstructionsXingwei He, Qianru Zhang, Pengfei Chen, Guanhua Chen 等AAAI 2026 · 2 次引用
- UltraIF: Advancing Instruction Following from the WildKaikai An, Li Sheng, Ganqu Cui, Shuzheng Si 等EMNLP 2025 · 1 次引用
- CFBench: A Comprehensive Constraints-Following Benchmark for LLMsTao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo 等ACL 2025 · 53 次引用
- MM-IFEngine: Towards Multimodal Instruction FollowingShengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang 等ICCV 2025 · 3 次引用
