EMNLP Findings2024
TOWER: Tree Organized Weighting for Evaluating Complex Instructions
Noah Ziems, Zhihan Zhang, Meng Jiang
Abstract
Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow.
To address this gap, we propose a novel evaluation metric, TOWER, that incorporates humanjudged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models cites WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models cites InFoBench: Evaluating Instruction Following Ability in Large Language Models
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild cites Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild cites InFoBench: Evaluating Instruction Following Ability in Large Language Models
Cited by top-tier papers1
Builds on6
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
- WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
- InFoBench: Evaluating Instruction Following Ability in Large Language ModelsYiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao et al.ACL Findings 2024 · 26 citations
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the WildBill Yuchen Lin, Yuntian Deng, Khyathi Raghavi Chandu, Abhilasha Ravichander et al.ICLR 2025
Related papers
- IF-RewardBench: Benchmarking Judge Models for Instruction-Following EvaluationBosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling et al.ACL 2026 · 3 citations
- Can Large Language Models Understand Real-World Complex Instructions?Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen et al.AAAI 2024 · 99 citations
- Evaluating Large Language Models at Evaluating Instruction FollowingZhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng et al.ICLR 2024 · 299 citations
- LexInstructEval: Lexical Instruction Following Evaluation for Large Language ModelsHuimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun et al.AAAI 2026
- StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction FollowingJinnan Li, Jinzhe Li, Yue Wang, Yi Chang et al.ACL Findings 2025 · 3 citations
